```python
import pyprob
from pyprob import Model
from pyprob.distributions import Normal

import torch
import numpy as np
import math
import matplotlib.pyplot as plt
%matplotlib inline
fig = plt.figure();
```

    <Figure size 432x288 with 0 Axes>

# Defining the model
First, we define the model as a probabilistic program inheriting from `pyprob.Model`. Models inherit from `torch.nn.Module` and can be potentially trained with gradient-based optimization (not covered in this example).

The `forward` function can have any number and type of arguments as needed.

```python
class GaussianUnknownMean(Model):
    def __init__(self):
        super().__init__(name='Gaussian with unknown mean') # give the model a name
        self.prior_mean = 1
        self.prior_std = math.sqrt(5)
        self.likelihood_std = math.sqrt(2)

    def forward(self): # Needed to specifcy how the generative model is run forward
        # sample the (latent) mean variable to be inferred:
        mu = pyprob.sample(Normal(self.prior_mean, self.prior_std)) # NOTE: sample -> denotes latent variables

        # define the likelihood
        likelihood = Normal(mu, self.likelihood_std)

        # Lets add two observed variables
        # -> the 'name' argument is used later to assignment values:
        pyprob.observe(likelihood, name='obs0') # NOTE: observe -> denotes observable variables
        pyprob.observe(likelihood, name='obs1')

        # return the latent quantity of interest
        return mu

model = GaussianUnknownMean()
```

# Finding the correct posterior analytically
Since all distributions are gaussians in this model, we can analytically compute the posterior and we can compare the true posterior to the inferenced one.

Assuming that the prior and likelihood are $p(x) = \mathcal{N}(\mu_0, \sigma_0)$ and $p(y|x) = \mathcal{N}(x, \sigma)$ respectively and, $y_1, y_2, \ldots y_n$ are the observed values, the posterior would be $p(x|y) = \mathcal{N}(\mu_p, \sigma_p)$ where,
$$
\begin{align}
\sigma_{p}^{2} & = \frac{1}{\frac{n}{\sigma^2} + \frac{1}{\sigma_{0}^{2}}} \\
\mu_p & = \sigma_{p}^{2} \left( \frac{\mu_0}{\sigma_{0}^{2}} + \frac{n\overline{y}}{\sigma^2} \right)
\end{align}
$$
The following class implements computing this posterior distribution. We also implement some helper functions and variables for plotting the correct posterior and prior.

```python
def plot_function(min_val, max_val, func, *args, **kwargs):
        x = np.linspace(min_val,max_val,int((max_val-min_val)*50))
        plt.plot(x, np.vectorize(func)(x), *args, **kwargs)

def get_dist_pdf(dist):
    return lambda x: math.exp(dist.log_prob(x))

class CorrectDistributions:
    def __init__(self, model):
        self.prior_mean = model.prior_mean
        self.prior_std = model.prior_std
        self.likelihood_std = model.likelihood_std
        self.prior_dist = Normal(self.prior_mean, self.prior_std)

    @property
    def observed_list(self):
        return self.__observed_list

    @observed_list.setter
    def observed_list(self, new_observed_list):
        self.__observed_list = new_observed_list
        self.construct_correct_posterior()

    def construct_correct_posterior(self):
        n = len(self.observed_list)
        posterior_var = 1/(n/self.likelihood_std**2 + 1/self.prior_std**2)
        posterior_mu = posterior_var * (self.prior_mean/self.prior_std**2 + n*np.mean(self.observed_list)/self.likelihood_std**2)
        self.posterior_dist = Normal(posterior_mu, math.sqrt(posterior_var))

    def prior_pdf(self, model, x):
        p = Normal(model.prior_mean,model.prior_stdd)
        return math.exp(p.log_prob(x))

    def plot_posterior(self, min_val, max_val):
        if not hasattr(self, 'posterior_dist'):
            raise AttributeError('observed values are not set yet, and posterior is not defined.')
        plot_function(min_val, max_val, get_dist_pdf(self.posterior_dist), label='correct posterior', color='orange')

    def plot_prior(self, min_val, max_val):
        plot_function(min_val, max_val, get_dist_pdf(self.prior_dist), label='prior', color='green')

correct_dists = CorrectDistributions(model)
```

# Prior distribution
We inspect the prior distribution to see if it behaves in the way we intended. First we construct an `Empirical` distribution with forward samples from the model.

Note: Extra arguments passed to `prior_distribution` will be forwarded to model's `forward` function.

```python
prior = model.prior_distribution(num_traces=1000)
```

    Time spent  | Time remain.| Progress             | Trace     | Traces/sec
    0d:00:00:00 | 0d:00:00:00 | #################### | 1000/1000 | 1,293.24

We can plot a historgram of these samples that are held by the `Empirical` distribution.

```python
prior.plot_histogram(show=False, alpha=0.75, label='emprical prior')
correct_dists.plot_prior(min(prior.values_numpy()),max(prior.values_numpy()))
plt.legend();
```

# Posterior inference with importance sampling
For a given set of observations, we can get samples from the posterior distribution.

```python
correct_dists.observed_list = [8, 9] # Observations
# sample from posterior (5000 samples)
posterior = model.posterior_distribution(
                                         num_traces=5000, # the number of samples estimating the posterior
                                         inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING, # specify which inference engine to use
                                         observe={'obs0': correct_dists.observed_list[0],
                                                  'obs1': correct_dists.observed_list[1]} # assign values to the observed values
                                         )
```

    Time spent  | Time remain.| Progress             | Trace     | Traces/sec
    0d:00:00:02 | 0d:00:00:00 | #################### | 5000/5000 | 1,840.04

Regular importance sampling uses proposals from the prior distribution. We can see this by plotting the histogram of the posterior distribution without using the importance weights. As expected, this is the same with the prior distribution.

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(show=False, alpha=0.75, label='empirical proposal')
correct_dists.plot_prior(min(posterior_unweighted.values_numpy()),
                         max(posterior_unweighted.values_numpy()))
correct_dists.plot_posterior(min(posterior_unweighted.values_numpy()),
                             max(posterior_unweighted.values_numpy()))
plt.legend();
```

When we do use the weights, we end up with the correct posterior distribution. The following shows the sampled posterior with the correct posterior (orange curve).

```python
posterior.plot_histogram(show=False, alpha=0.75, bins=50, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior_unweighted.values_numpy()))
plt.legend();
```

In practice, it is advised to use methods of the `Empirical` posterior distribution instead of dealing with the weights directly, which ensures that the weights are used in the correct way.

For instance, we can get samples from the posterior, compute its mean and standard deviation, and evaluate expectations of a function under the distribution:

```python
print(posterior.sample())
```

    tensor(7.0136)

```python
print(posterior.mean)
```

    tensor(7.1378)

```python
print(posterior.stddev)
```

    tensor(0.9106)

```python
print(posterior.expectation(lambda x: torch.sin(x)))
```

    tensor(0.4846)

# Inference compilation
Inference compilation is a technique where a deep neural network is used for parameterizing the proposal distribution in importance sampling (https://arxiv.org/abs/1610.09900). This neural network, which we call inference network, is automatically generated and trained with data sampled from the model.

We can learn an inference network for our model.

```python
model.learn_inference_network(num_traces=20000,
                              observe_embeddings={'obs0' : {'dim' : 32},
                                                  'obs1': {'dim' : 32}},
                              inference_network=pyprob.InferenceNetwork.LSTM)
```

    Creating new inference network...
    Observable obs0: observe embedding not specified, using the default FEEDFORWARD.
    Observable obs0: embedding depth not specified, using the default 2.
    Observable obs1: observe embedding not specified, using the default FEEDFORWARD.
    Observable obs1: embedding depth not specified, using the default 2.
    Observe embedding dimension: 64
    Train. time | Epoch| Trace     | Init. loss| Min. loss | Curr. loss| T.since min | Traces/sec
    New layers, address: 16__forward__mu__Normal__1, distribution: Normal
    Total addresses: 1, distribution types: 1, parameters: 1,643,583
    0d:00:00:33 | 1    | 20,032    | +2.40e+00 | +1.10e+00 | +1.26e+00 | 0d:00:00:05 | 1,003.0

We now construct the posterior distribution using samples from inference compilation, using the trained inference network.

A much smaller number of samples are enough (200 vs. 5000) because the inference network provides good proposals based on the given observations. We can see that the proposal distribution given by the inference network is doing a job much better than the prior, by plotting the posterior samples without the importance weights, for a selection of observations.

```python
# sample from posterior (200 samples)
posterior = model.posterior_distribution(
                                         num_traces=200, # the number of samples estimating the posterior
                                         inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING_WITH_INFERENCE_NETWORK, # specify which inference engine to use
                                         observe={'obs0': correct_dists.observed_list[0],
                                                  'obs1': correct_dists.observed_list[1]} # assign values to the observed values
                                         )
```

    Time spent  | Time remain.| Progress             | Trace   | Traces/sec
    0d:00:00:00 | 0d:00:00:00 | #################### | 200/200 | 335.57

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(show=False, bins=50, alpha=0.75, label='empirical proposal')

correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior.values_numpy()))
plt.legend();
```

We can see that the proposal distribution given by the inference network is already a good estimate to the true posterior which makes the inferred posterior a much better estimate than the prior, even using much less number of samples.

```python
posterior.plot_histogram(show=False, bins=50, alpha=0.75, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                             max(posterior.values_numpy()))
plt.legend();
```

Inference compilation performs amortized inferece which means, the same trained network provides proposal distributions for any observed values.

We can try performing inference using the same trained network with different observed values.

```python
correct_dists.observed_list = [12, 10] # New observations

posterior = model.posterior_distribution(
                                         num_traces=200,
                                         inference_engine=pyprob.InferenceEngine.IMPORTANCE_SAMPLING_WITH_INFERENCE_NETWORK, # specify which inference engine to use
                                         observe={'obs0': correct_dists.observed_list[0],
                                                  'obs1': correct_dists.observed_list[1]}
                                         )
```

    Time spent  | Time remain.| Progress             | Trace   | Traces/sec
    0d:00:00:00 | 0d:00:00:00 | #################### | 200/200 | 273.36

```python
posterior_unweighted = posterior.unweighted()
posterior_unweighted.plot_histogram(show=False, bins=50, alpha=0.75, label='empirical proposal')

correct_dists.plot_posterior(min(posterior.values_numpy()),
                       max(posterior.values_numpy()))
plt.legend();
```

```python
posterior.plot_histogram(show=False, bins=50, alpha=0.75, label='inferred posterior')
correct_dists.plot_posterior(min(posterior.values_numpy()),
                       max(posterior.values_numpy()))
plt.legend();
```


# Application of 2nd order Runge Kutta to Populations Equations

This notebook implements the 2nd Order Runge Kutta method for three different population intial value problems.

# 2nd Order Runge Kutta
The general 2nd Order Runge Kutta method for to the first order differential equation
\begin{equation} y^{'} = f(t,y), \end{equation}
numerical approximates $y$ the at time point $t_i$ as $w_i$
with the  formula:
\begin{equation} w_{i+1}=w_i+\frac{h}{2}\big[k_1+k_2],\end{equation}
for $i=0,...,N-1$, where
\begin{equation}k_1=f(t_i,w_i)\end{equation}
and
\begin{equation}k_2=f(t_i+h,w_i+hk_1)\end{equation}
and $h$ is the stepsize.

To illustrate the method we will apply it to three intial value problems:
## 1. Linear
Consider the linear population Differential Equation
\begin{equation} y^{'}=0.1y, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## 2. Non-Linear Population Equation
Consider the non-linear population Differential Equation
\begin{equation} y^{'}=0.2y-0.01y^2, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## 3. Non-Linear Population Equation with an oscillation
Consider the non-linear population Differential Equation with an oscillation
\begin{equation} y^{'}=0.2y-0.01y^2+\sin(2\pi t), \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

#### Setting up Libraries

```python
## Library
import numpy as np
import math
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

## Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
\begin{equation} h=\frac{b-a}{N}.\end{equation}
Here the interval is $2000\leq t \leq 2020,$
\begin{equation} h=\frac{2020-2000}{200}=0.1.\end{equation}
This gives the 201 discrete points:
\begin{equation} t_0=2000, \ t_1=2000.1, \ ... t_{200}=2020. \end{equation}
This is generalised to
\begin{equation} t_i=2000+i0.1, \ \ \ i=0,1,...,200.\end{equation}
The plot below shows the discrete time steps:

```python
N=200
t_end=2020.0
t_start=2000.0
h=((t_end-t_start)/N)
t=np.arange(t_start,t_end+h/2,h)
fig = plt.figure(figsize=(10,4))
plt.plot(t,0*t,'o:',color='red')
plt.title('Illustration of discrete time points for h=%s'%(h))
plt.show()
```

# 1. Linear Population Equation
## Exact Solution
The linear population equation
\begin{equation}y^{'}=0.1y, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}
has a known exact (analytic) solution
\begin{equation} y=6e^{0.1(t-2000)}. \end{equation}

## Specific 2nd Order Runge Kutta
To write the specific 2nd Order Runge Kutta method for the linear population equation we need
\begin{equation}f(t,y)=0.1y.\end{equation}

```python
def linfun(t,w):
    ftw=0.1*w
    return ftw
```

this gives
\begin{equation}k_1=f(t_i,w_i)=0.lw_i,\end{equation}
\begin{equation}k_2=f(t_i+h,w_i+hk_1)=0.1(w_i+hk_1),\end{equation}
and the difference equation
\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}(k_1+k_2)\end{equation}
for $i=0,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with step size $h$ and the initial condition
\begin{equation}w_0=6.\end{equation}

```python
w=np.zeros(N+1)
w[0]=6.0
## 2nd Order Runge Kutta
for k in range (0,N):
    k1=linfun(t[k],w[k])
    k2=linfun(t[k]+h,w[k]+h*k1)
    w[k+1]=w[k]+h/2*(k1+k2)
```

## Plotting Results

```python
y=6*np.exp(0.1*(t-2000))
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Runge Kutta')
plt.plot(t,y,'s:',color='black',label='Exact')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time, the Runge Kutta numerical approximation, $w$,  the exact solution, $y$, and the exact error $|y(t_i)-w_i|$ for the linear population equation:

```python

d = {'time t_i': t[0:10],    'Runge Kutta':w[0:10],'Exact (y)':y[0:10],'Exact Error':np.abs(np.round(y[0:10]-w[0:10],10))}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Runge Kutta
      Exact (y)
      Exact Error

      0
      2000.0
      6.000000
      6.000000
      0.000000

      1
      2000.1
      6.060300
      6.060301
      0.000001

      2
      2000.2
      6.121206
      6.121208
      0.000002

      3
      2000.3
      6.182724
      6.182727
      0.000003

      4
      2000.4
      6.244861
      6.244865
      0.000004

      5
      2000.5
      6.307621
      6.307627
      0.000005

      6
      2000.6
      6.371013
      6.371019
      0.000006

      7
      2000.7
      6.435042
      6.435049
      0.000007

      8
      2000.8
      6.499714
      6.499722
      0.000009

      9
      2000.9
      6.565036
      6.565046
      0.000010

## 2. Non-Linear Population Equation
\begin{equation} y^{'}=0.2y-0.01y^2, \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}
## Specific 2nd Order Runge Kutta for the Non-Linear Population Equation
To write the specific 2nd Order Runge Kutta method we need
\begin{equation}f(t,y)=0.2y-0.01y^2,\end{equation}
this gives
\begin{equation}k_1=f(t_i,w_i)=0.2w_i-0.01w_i^2,\end{equation}
\begin{equation}k_2=f(t_i+h,w_i+hk_1)=0.2(w_i+hk_1)-0.01(w_i+hk_1)^2,\end{equation}
and the difference equation
\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}(k_1+k_2)\end{equation}
for $i=0,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with step size $h$ and the initial condition
\begin{equation}w_0=6.\end{equation}

```python
def nonlinfun(t,w):
    ftw=0.2*w-0.01*w*w
    return ftw
```

```python
w=np.zeros(N+1)
w[0]=6.0
## 2nd Order Runge Kutta
for k in range (0,N):
    k1=nonlinfun(t[k],w[k])
    k2=nonlinfun(t[k]+h,w[k]+h*k1)
    w[k+1]=w[k]+h/2*(k1+k2)
```

## Results
The plot below shows the Runge Kutta numerical approximation, $w$ (circles) for the non-linear population equation:

```python
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Runge Kutta')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time and the Runge Kutta numerical approximation, $w$,  for the non-linear population equation:

```python
d = {'time t_i': t[0:10],
     'Runge Kutta':w[0:10]}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Runge Kutta

      0
      2000.0
      6.000000

      1
      2000.1
      6.084332

      2
      2000.2
      6.169328

      3
      2000.3
      6.254977

      4
      2000.4
      6.341270

      5
      2000.5
      6.428197

      6
      2000.6
      6.515747

      7
      2000.7
      6.603909

      8
      2000.8
      6.692672

      9
      2000.9
      6.782025

## 3. Non-Linear Population Equation with an oscilation
\begin{equation} y^{'}=0.2y-0.01y^2+\sin(2\pi t), \ \ (2000 \leq t \leq 2020), \end{equation}
with the initial condition,
\begin{equation}y(2000)=6.\end{equation}

## Specific 2nd Order Runge Kutta for the Non-Linear Population Equation with an oscilation
To write the specific 2nd Order Runge Kutta difference equation for the intial value problem we need
\begin{equation}f(t,y)=0.2y-0.01y^2+\sin(2\pi t),\end{equation}
which gives
\begin{equation}k_1=f(t_i,w_i)=0.2w_i-0.01w_i^2+\sin(2\pi t_i),\end{equation}
\begin{equation}k_2=f(t_i+h,w_i+hk_1)=0.2(w_i+hk_1)-0.01(w_i+hk_1)^2+\sin(2\pi (t_i+h)),\end{equation}
and the difference equation
\begin{equation}w_{i+1}=w_{i}+\frac{h}{2}(k_1+k_2)\end{equation}
for $i=0,...,199$, where $w_i$ is the numerical approximation of $y$ at time $t_i$, with step size $h$ and the initial condition
\begin{equation}w_0=6.\end{equation}

```python
def nonlin_oscfun(t,w):
    ftw=0.2*w-0.01*w*w+np.sin(2*np.math.pi*t)
    return ftw
```

```python
w=np.zeros(N+1)
w[0]=6.0
## 2nd Order Runge Kutta
for k in range (0,N):
    k1=nonlin_oscfun(t[k],w[k])
    k2=nonlin_oscfun(t[k]+h,w[k]+h*k1)
    w[k+1]=w[k]+h/2*(k1+k2)
```

## Results
The plot below shows the 2nd order Runge Kutta numerical approximation, $w$ (circles) for the non-linear population equation:

```python
fig = plt.figure(figsize=(8,4))
plt.plot(t,w,'o:',color='purple',label='Taylor')
plt.legend(loc='best')
plt.show()
```

## Table
The table below shows the time and the 2nd order Runge Kutta numerical approximation, $w$,  for the non-linear population equation:

```python
d = {'time t_i': t[0:10],
     'Runge Kutta':w[0:10]}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Runge Kutta

      0
      2000.0
      6.000000

      1
      2000.1
      6.113722

      2
      2000.2
      6.276109

      3
      2000.3
      6.458005

      4
      2000.4
      6.623032

      5
      2000.5
      6.741504

      6
      2000.6
      6.801784

      7
      2000.7
      6.814712

      8
      2000.8
      6.809444

      9
      2000.9
      6.822305

```python

```

Chapter 03 - Runge Kutta/Supplementary/301_Problem_Sheet.ipynb


# Binomial Distribution

> ***GitHub***: https://github.com/czs108

## Definition

\begin{equation}
P(X = r) =\, ^{n}C_{r} \cdot p^{r} \cdot (1 - p)^{n - r}
\end{equation}

\begin{align}
X &= \text{The total number of successes.} \\
p &= \text{The probability of a success on an individual trial.} \\
n &= \text{The number of trials.}
\end{align}

If a variable $X$ follows a *Binomial Distribution* where the probability of *success* in a trial is $p$, and $n$ is the *total* number of trials. This can be written as

\begin{equation}
X \sim B(n,\, p)
\end{equation}

## Expectation

\begin{equation}
E(X) = np
\end{equation}

## Variance

\begin{equation}
Var(X) = np \cdot (1 - p)
\end{equation}

## Approximation

### Poisson Distribution

$X \sim B(n,\, p)$ can be approximated by the *Poisson Distribution* $X \sim Po(np)$ if $n$ is *large* and $p$ is *small*.

Because both the *expectation* and *variance* of $X \sim Po(np)$ are $np$. When $n$ is large and $p$ is small, $(1 - p) \approx 1$ and for $X \sim B(n,\, p)$:

\begin{equation}
E(X) = np
\end{equation}

\begin{equation}
Var(X) \approx np
\end{equation}

The approximation is typically very close if $n > 50$ and $p < 0.1$.

### Normal Distribution

Suppose $q$ is $(1 - p)$.

Normally if $np > 5$ and $nq > 5$, $X \sim B(n,\, p)$ also can be approximated by the *Normal Distribution* $X \sim N(np,\, npq)$.


```python

```

# Linear Regression

- Dataset consists of 2 types of variables
    - Independent variables / features / decsriptors / input (random) variables / covariates / regressor
    - Dependent variables / output / class / target

Regression is a method of modelling a target value based on independent variables.

Linear regression is a method to find the line of best fit given a dataset so that we can predict one variable given the other

$
\begin{align}
y = mx + c
\end{align}
$

where m is the slope, c is the y intercept

## Read the data from input file - Using [Pandas](https://pandas.pydata.org/)

- Generally the data is provided in the tabular format in the form of .csv, .xls or via a database.
- Pandas is predominantly used for data wrangling and analysis for a dataset in a tabular format

```python
import numpy as np
from matplotlib import pyplot as plt

fig = plt.figure(figsize=(16, 6))

m = 3
c = 7

x_vals = [i for i in range(-5, 6)]
y_vals = [m*x_ + c for x_ in x_vals]

ax = fig.gca()
ax.set_xticks(np.arange(min(x_vals)-1, max(x_vals)+1, 1))
ax.set_yticks(np.arange(min(y_vals)-1, max(y_vals)+1, 1))
plt.subplot(1,2,1)
plt.plot(x_vals, y_vals, color='r')
plt.title('Positive Slope')
plt.grid()

m = -3
c = 7

x_vals = [i for i in range(-5, 6)]
y_vals = [m*x_ + c for x_ in x_vals]

ax.set_xticks(np.arange(min(x_vals)-1, max(x_vals)+1, 1))
ax.set_yticks(np.arange(min(y_vals)-1, max(y_vals)+1, 1))
plt.subplot(1,2,2)
plt.plot(x_vals, y_vals, color='r')
plt.title('Negative Slope')
plt.grid()

plt.show()
```

```python
import pandas as pd
df_data = pd.read_csv('../data/2d_classification.csv')
```

```python
from matplotlib import pyplot as plt

plt.rcParams['figure.figsize'] = [10, 7] # Size of the plots

colors = {0:'b', 1:'g', 2:'r', 3:'c', 4:'m', 5:'y', 6:'k'}
fig, ax = plt.subplots()
grouped = df_data.groupby('label')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='x', y='y', label=key, color=colors[key])
plt.show()
```

```python
from sklearn.linear_model import LinearRegression

x = df_data[['x']].values
y = df_data[['y']].values

lin_regr = LinearRegression()
lin_regr.fit(x, y)

pred_y = lin_regr.predict(x)
```

```python
colors = {0:'b', 1:'g', 2:'r', 3:'c', 4:'m', 5:'y', 6:'k'}

plt.figure(figsize=(8,6))
plt.scatter(x, y,  color='black')
plt.plot(x, pred_y, color='blue', linewidth=1)
plt.xlabel('X')
plt.ylabel('Y')
plt.title('Linear Regression')
plt.show()
```

## Components to run an algorithm on sklearn

### Step 1: Instance of the algorithm that is to be used
- Create an instance of the algorithm that could be invoked to use the implemented functions

 __lin_regr = LinearRegression()__

### Step 2: Train your data -> Fit function

- What does it do ?
    - It trains on the dataset
- What does it identify ?
    - Coefficients / Weights / Intercepts
- When will it be used ?
    - Use the co-efficients to predict for the new data

 __lin_regr.fit(x, y)__

### Step 3: Prediction
- For any new set of data, you can predict using

 __pred_y = lin_regr.predict(x)__

### What does your model store ?

#### Coeficients

```python
print('Coefficients:', lin_regr.coef_)
```

    Coefficients: [[-0.33526649]]

#### Intercepts

```python
print('Intercepts:', lin_regr.intercept_)
```

    Intercepts: [7.86368678]

```python
import numpy as np
from matplotlib import pyplot as plt

fig = plt.figure(figsize=(8, 6))

m = lin_regr.coef_[0]
c = lin_regr.intercept_

x_vals = [i for i in range(-10, 3)]
y_vals = [m*x_ + c for x_ in x_vals]

ax.set_xticks(np.arange(min(x_vals)-1, max(x_vals)+1, 1))
ax.set_yticks(np.arange(min(y_vals)-1, max(y_vals)+1, 1))
plt.subplot(1,1,1)
plt.plot(x_vals, y_vals, color='r')
plt.title('Negative Slope')
plt.grid()

plt.show()
```

## Metrics

#### Mean Suqared Error

```python
from sklearn.metrics import mean_squared_error
print('Mean squared error: %.2f' % mean_squared_error(y, pred_y))
```

    Mean squared error: 0.95

#### R^2 Score
- proportion of the variance in the dependent variable(target/label) that is predictable from the independent variables(features)

```python
from sklearn.metrics import r2_score
print('Variance score: %.2f' % r2_score(y, pred_y))
```

    Variance score: 0.22

### Another example

```python
from sklearn.datasets.samples_generator import make_blobs
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.linear_model import LinearRegression

# Creating data
data, label = make_blobs(n_samples=100, centers=1, n_features=2)
df_data = pd.DataFrame(dict(x=data[:,0], y=data[:,1], label=label)) # converting it into dataframe to make it easier

# Making them easy to read
x = df_data[['x']].values
y = df_data[['y']].values

print(type(x))
print(x.shape)
# Prediction
lin_regr = LinearRegression()
lin_regr.fit(x, y)
pred_y = lin_regr.predict(x)

# Plotting the results
plt.figure()
plt.scatter(x, y,  color='black')
plt.plot(x, pred_y, color='blue', linewidth=1)
plt.show()
```

### Restrictions of Linear Regression

```python
from matplotlib import pyplot as plt

data = np.array([[1,0], [2,0], [3,0], [4,0], [5,0], [6,1], [7,1], [8,1], [9,1], [10, 1]])

x = np.reshape(data[:, 0], (data.shape[0], 1))
y =  np.reshape(data[:, 1], (data.shape[0], 1))

# Prediction
lin_regr = LinearRegression()
lin_regr.fit(x, y)
pred_y = lin_regr.predict(x)

# Plotting the results
plt.figure(figsize=(10, 8))
plt.scatter(x, y,  color='black')
plt.plot(x, pred_y, color='blue', linewidth=1)
plt.axhline(y=0.5, xmin=0, xmax=1, color='m')
plt.axvline(x=5.5, ymin=0, ymax=1, color='g', dashes=[3,3])
plt.show()
```

```python
from matplotlib import pyplot as plt

data = np.array([[1,0], [2,0], [3,0], [4,0], [5,0], [6,1], [7,1], [8,1], [9,1], [10, 1], [14, 1], [16,1], [17, 1]])

x = np.reshape(data[:, 0], (data.shape[0], 1))
y =  np.reshape(data[:, 1], (data.shape[0], 1))

# Prediction
lin_regr = LinearRegression()
lin_regr.fit(x, y)
pred_y = lin_regr.predict(x)

# Plotting the results
plt.figure(figsize=(10, 8))
plt.scatter(x, y,  color='black')
plt.plot(x, pred_y, color='blue', linewidth=1)
plt.axhline(y=0.5, xmin=0, xmax=1, color='m')
plt.axvline(x=6.25, ymin=0, ymax=1, color='g', dashes=[3,3])
plt.show()
```

classification/notebooks/.ipynb_checkpoints/02 - Linear Regression-checkpoint.ipynb


In school, students are taught to draw lines like the following.

$$ y = 2 x + 1$$
They're taught to pick two values for $x$ and calculate the corresponding values for $y$ using the equation. Then they draw a set of axes, plot the points, and then draw a line extending through the two dots on their axes.

```python
# Import matplotlib.
import matplotlib.pyplot as plt
```

```python
# Draw some axes.
plt.plot([-1, 10], [0, 0], 'k-')
plt.plot([0, 0], [-1, 10], 'k-')

# Plot the red, blue and green lines.
plt.plot([1, 1], [-1, 3], 'b:')
plt.plot([-1, 1], [3, 3], 'r:')

# Plot the two points (1,3) and (2,5).
plt.plot([1, 2], [3, 5], 'ko')
# Join them with an (extending) green lines.
plt.plot([-1, 10], [-1, 21], 'g-')

# Set some reasonable plot limits.
plt.xlim([-1, 10])
plt.ylim([-1, 10])

# Show the plot.
plt.show()
```

```python
Simple linear regression is about the opposite problem - what if you have some points and are looking for the equation? It's easy when the points are perfectly on a line already, but usually real-world data has some noise. The data might still look roughly linear, but aren't exactly so.
```

```python
Example (contrived and simulated)
```

Scenario
Suppose you are trying to weigh your suitcase to avoid an airline's extra charges. You don't have a weighing scales, but you do have a spring and some gym-style weights of masses 7KG, 14KG and 21KG. You attach the spring to the wall hook, and mark where the bottom of it hangs. You then hang the 7KG weight on the end and mark where the bottom of the spring is. You repeat this with the 14KG weight and the 21KG weight. Finally, you place your case hanging on the spring, and the spring hangs down halfway between the 7KG mark and the 14KG mark. Is your case over the 10KG limit set by the airline?

Hypothesis
When you look at the marks on the wall, it seems that the 0KG, 7KG, 14KG and 21KG marks are evenly spaced. You wonder if that means your case weighs 10.5KG. That is, you wonder if there is a linear relationship between the distance the spring's hook is from its resting position, and the mass on the end of it.

Experiment
You decide to experiment. You buy some new weights - a 1KG, a 2KG, a 3Kg, all the way up to 20KG. You place them each in turn on the spring and measure the distance the spring moves from the resting position. You tabulate the data and plot them.

Analysis
Here we'll import the Python libraries we need for or investigations below

```python

# Make matplotlib show interactive plots in the notebook.
%matplotlib inline

```

```python
# numpy efficiently deals with numerical multi-dimensional arrays.
import numpy as np

# matplotlib is a plotting library, and pyplot is its easy-to-use module.
import matplotlib.pyplot as plt

# This just sets the default plot size to be bigger.
plt.rcParams['figure.figsize'] = (8, 6)
```

```python
Ignore the next couple of lines where I fake up some data. I'll use the fact that I faked the data to explain some results later. Just pretend that w is an array containing the weight values and d are the corresponding distance measurements.
```

```python
w = np.arange(0.0, 21.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)
```

```python

# Let's have a look at w.
w
```

```python
# Let's have a look at d.
d
```

```python
# Look at the data from the experiment
# Create the plot.

plt.plot(w, d, 'k.')

# Set some properties for the plot.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')

# Show the plot.
plt.show()
```

```python

Model
It looks like the data might indeed be linear. The points don't exactly fit on a straight line, but they are not far off it. We might put that down to some other factors, such as the air density, or errors, such as in our tape measure. Then we can go ahead and see what would be the best line to fit the data.

Straight lines
All straight lines can be expressed in the form $y = mx + c$. The number $m$ is the slope of the line. The slope is how much $y$ increases by when $x$ is increased by 1.0. The number $c$ is the y-intercept of the line. It's the value of $y$ when $x$ is 0.

Fitting the model
To fit a straight line to the data, we just must pick values for $m$ and $c$. These are called the parameters of the model, and we want to pick the best values possible for the parameters. That is, the best parameter values given the data observed. Below we show various lines plotted over the data, with different values for $m$ and $c$.
```

```python
# Plot w versus d with black dots.
plt.plot(w, d, 'k.', label="Data")

# Overlay some lines on the plot.
x = np.arange(0.0, 21.0, 1.0)
plt.plot(x, 5.0 * x + 10.0, 'r-', label=r"$5x + 10$")
plt.plot(x, 6.0 * x +  5.0, 'g-', label=r"$6x +  5$")
plt.plot(x, 5.0 * x + 15.0, 'b-', label=r"$5x + 15$")

# Add a legend.
plt.legend()

# Add axis labels.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')

# Show the plot.
plt.show()
```

```python
Calculating the cost
You can see that each of these lines roughly fits the data. Which one is best, and is there another line that is better than all three? Is there a "best" line?

It depends how you define the word best. Luckily, everyone seems to have settled on what the best means. The best line is the one that minimises the following calculated value.

$$ \sum_i (y_i - mx_i - c)^2 $$
Here $(x_i, y_i)$ is the $i^{th}$ point in the data set and $\sum_i$ means to sum over all points. The values of $m$ and $c$ are to be determined. We usually denote the above as $Cost(m, c)$.

Where does the above calculation come from? It's easy to explain the part in the brackets $(y_i - mx_i - c)$. The corresponding value to $x_i$ in the dataset is $y_i$. These are the measured values. The value $m x_i + c$ is what the model says $y_i$ should have been. The difference between the value that was observed ($y_i$) and the value that the model gives ($m x_i + c$), is $y_i - mx_i - c$.

Why square that value? Well note that the value could be positive or negative, and you sum over all of these values. If we allow the values to be positive or negative, then the positive could cancel the negatives. So, the natural thing to do is to take the absolute value $\mid y_i - m x_i - c \mid$. Well it turns out that absolute values are a pain to deal with, and instead it was decided to just square the quantity instead, as the square of a number is always positive. There are pros and cons to using the square instead of the absolute value, but the square is used. This is usually called least squares fitting.
```

```python
# Calculate the cost of the lines above for the data above.
cost = lambda m,c: np.sum([(d[i] - m * w[i] - c)**2 for i in range(w.size)])

print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (5.0, 10.0, cost(5.0, 10.0)))
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (6.0,  5.0, cost(6.0,  5.0)))
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (5.0, 15.0, cost(5.0, 15.0)))
```

```python
Cost with m =  5.00 and c = 10.00:   364.91
Cost with m =  6.00 and c =  5.00:  1911.26
Cost with m =  5.00 and c = 15.00:   784.03
Minimising the cost
We want to calculate values of $m$ and $c$ that give the lowest value for the cost value above. For our given data set we can plot the cost value/function. Recall that the cost is:

$$ Cost(m, c) = \sum_i (y_i - mx_i - c)^2 $$
This is a function of two variables, $m$ and $c$, so a plot of it is three dimensional. See the Advanced section below for the plot.

In the case of fitting a two-dimensional line to a few data points, we can easily calculate exactly the best values of $m$ and $c$. Some of the details are discussed in the Advanced section, as they involve calculus, but the resulting code is straight-forward. We first calculate the mean (average) values of our $x$ values and that of our $y$ values. Then we subtract the mean of $x$ from each of the $x$ values, and the mean of $y$ from each of the $y$ values. Then we take the dot product of the new $x$ values and the new $y$ values and divide it by the dot product of the new $x$ values with themselves. That gives us $m$, and we use $m$ to calculate $c$.

Remember that in our dataset $x$ is called $w$ (for weight) and $y$ is called $d$ (for distance). We calculate $m$ and $c$ below.
```

```python
# Calculate the best values for m and c.

# First calculate the means (a.k.a. averages) of w and d.
w_avg = np.mean(w)
d_avg = np.mean(d)

# Subtract means from w and d.
w_zero = w - w_avg
d_zero = d - d_avg

# The best m is found by the following calculation.
m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
# Use m from above to calculate the best c.
c = d_avg - m * w_avg

print("m is %8.6f and c is %6.6f." % (m, c))
```

```python
#Note that numpy has a function that will perform this calculation for us, called polyfit. It can be used to fit lines in many dimensions.
np.polyfit(w, d, 1)
```

```python
Best fit line¶
So, the best values for $m$ and $c$ given our data and using least squares fitting are about $4.95$ for $m$ and about $11.13$ for $c$. We plot this line on top of the data below.
```

```python
# Plot the best fit line.
plt.plot(w, d, 'k.', label='Original data')
plt.plot(w, m * w + c, 'b-', label='Best fit line')

# Add axis labels and a legend.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')
plt.legend()

# Show the plot.
plt.show()
```

```python
Note that the $Cost$ of the best $m$ and best $c$ is not zero in this case.
```

```python
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (m, c, cost(m, c)))

```

```python
Summary¶
In this notebook we:

Investigated the data.
Picked a model.
Picked a cost function.
Estimated the model parameter values that minimised our cost function.
Advanced
In the following sections we cover some of the more advanced concepts involved in fitting the line.

Simulating data
Earlier in the notebook we glossed over something important: we didn't actually do the weighing and measuring - we faked the data. A better term for this is simulation, which is an important tool in research, especially when testing methods such as simple linear regression.

We ran the following two commands to do this:

w = np.arange(0.0, 21.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)
The first command creates a numpy array containing all values between 1.0 and 21.0 (including 1.0 but not including 21.0) in steps of 1.0.
```

```python
np.arange(0.0, 21.0, 1.0)
```

```python
# The second command is more complex. First it takes the values in the w array, multiplies each by 5.0 and then adds 10.0.
5.0 * w + 10.0
```

```python
np.random.normal(0.0, 5.0, w.size)
```

```python
#The normal distribution follows a bell shaped curve. The curve is centred on the mean (0.0 in this case) and its general width is determined by the standard deviation (5.0 in this case).
# Plot the normal distrution.
normpdf = lambda mu, s, x: (1.0 / (2.0 * np.pi * s**2)) * np.exp(-((x - mu)**2)/(2 * s**2))

x = np.linspace(-20.0, 20.0, 100)
y = normpdf(0.0, 5.0, x)
plt.plot(x, y)

plt.show()
```

```python

The idea here is to add a little bit of randomness to the measurements of the distance. The random values are entered around 0.0, with a greater than 99% chance they're within the range -15.0 to 15.0. The normal distribution is used because of the Central Limit Theorem which basically states that when a bunch of random effects happen together the outcome looks roughly like the normal distribution. (Don't quote me on that!)

Plotting the cost function
We can plot the cost function for a given set of data points. Recall that the cost function involves two variables: $m$ and $c$, and that it looks like this:

$$ Cost(m,c) = \sum_i (y_i - mx_i - c)^2 $$
To plot a function of two variables we need a 3D plot. It can be difficult to get the viewing angle right in 3D plots, but below you can just about make out that there is a low point on the graph around the $(m, c) = (\approx 5.0, \approx 10.0)$ point.
```

```python
# This code is a little bit involved - don't worry about it.
# Just look at the plot below.

from mpl_toolkits.mplot3d import Axes3D

# Ask pyplot a 3D set of axes.
ax = plt.figure().gca(projection='3d')

# Make data.
mvals = np.linspace(4.5, 5.5, 100)
cvals = np.linspace(0.0, 20.0, 100)

# Fill the grid.
mvals, cvals = np.meshgrid(mvals, cvals)

# Flatten the meshes for convenience.
mflat = np.ravel(mvals)
cflat = np.ravel(cvals)

# Calculate the cost of each point on the grid.
C = [np.sum([(d[i] - m * w[i] - c)**2 for i in range(w.size)]) for m, c in zip(mflat, cflat)]
C = np.array(C).reshape(mvals.shape)

# Plot the surface.
surf = ax.plot_surface(mvals, cvals, C)

# Set the axis labels.
ax.set_xlabel('$m$', fontsize=16)
ax.set_ylabel('$c$', fontsize=16)
ax.set_zlabel('$Cost$', fontsize=16)

# Show the plot.
plt.show()
```

```python

Coefficient of determination
Earlier we used a cost function to determine the best line to fit the data. Usually the data do not perfectly fit on the best fit line, and so the cost is greater than 0. A quantity closely related to the cost is the coefficient of determination, also known as the R-squared value. The purpose of the R-squared value is to measure how much of the variance in $y$ is determined by $x$.

For instance, in our example the main thing that affects the distance the spring is hanging down is the weight on the end. It's not the only thing that affects it though. The room temperature and density of the air at the time of measurment probably affect it a little. The age of the spring, and how many times it has been stretched previously probably also have a small affect. There are probably lots of unknown factors affecting the measurment.

The R-squared value estimates how much of the changes in the $y$ value is due to the changes in the $x$ value compared to all of the other factors affecting the $y$ value. It is calculated as follows:

$$ R^2 = 1 - \frac{\sum_i (y_i - m x_i - c)^2}{\sum_i (y_i - \bar{y})^2} $$
Note that sometimes the Pearson correlation coefficient is used instead of the R-squared value. You can just square the Pearson coefficient to get the R-squred value.
```

```python
# Calculate the R-squared value for our data set.
rsq = 1.0 - (np.sum((d - m * w - c)**2)/np.sum((d - d_avg)**2))

print("The R-squared value is %6.4f" % rsq)
```

```python
# The same value using numpy.
np.corrcoef(w, d)[0][1]**2

```

The minimisation calculations
Earlier we used the following calculation to calculate $m$ and $c$ for the line of best fit. The code was:

w_zero = w - np.mean(w)
d_zero = d - np.mean(d)

m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
c = np.mean(d) - m * np.mean(w)
In mathematical notation we write this as:

$$ m = \frac{\sum_i (x_i - \bar{x}) (y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \qquad \textrm{and} \qquad c = \bar{y} - m \bar{x} $$
where $\bar{x}$ is the mean of $x$ and $\bar{y}$ that of $y$.

Where did these equations come from? They were derived using calculus. We'll give a brief overview of it here, but feel free to gloss over this section if it's not for you. If you can understand the first part, where we calculate the partial derivatives, then great!

The calculations look complex, but if you know basic differentiation, including the chain rule, you can easily derive them. First, we differentiate the cost function with respect to $m$ while treating $c$ as a constant, called a partial derivative. We write this as $\frac{\partial m}{ \partial Cost}$, using $\delta$ as opposed to $d$ to signify that we are treating the other variable as a constant. We then do the same with respect to $c$ while treating $m$ as a constant. We set both equal to zero, and then solve them as two simultaneous equations in two variables.

Calculate the partial derivatives
$$
\begin{align}
Cost(m, c)                       &= \sum_i (y_i - mx_i - c)^2      \\[1cm]
\frac{\partial Cost}{\partial m} &= \sum 2(y_i - m x_i -c)(-x_i) \\
                                 &= -2 \sum x_i (y_i - m x_i -c) \\[0.5cm]
\frac{\partial Cost}{\partial c} & = \sum 2(y_i - m x_i -c)(-1) \\
                                 & = -2 \sum (y_i - m x_i -c)   \\
\end{align}
$$
Set to zero
$$
\begin{align}
& \frac{\partial Cost}{\partial m}                         = 0 \\[0.2cm]
& \Rightarrow -2 \sum x_i (y_i - m x_i -c)                 = 0 \\
& \Rightarrow \sum (x_i y_i - m x_i x_i - x_i c)           = 0 \\
& \Rightarrow \sum x_i y_i - \sum_i m x_i x_i - \sum x_i c = 0 \\
& \Rightarrow m \sum x_i x_i                               = \sum x_i y_i - c \sum x_i \\[0.2cm]
& \Rightarrow m                                            = \frac{\sum x_i y_i - c \sum x_i}{\sum x_i x_i} \\[0.5cm]
& \frac{\partial Cost}{\partial c}               = 0 \\[0.2cm]
& \Rightarrow -2 \sum (y_i - m x_i - c)          = 0 \\
& \Rightarrow \sum y_i - \sum_i m x_i - \sum c   = 0 \\
& \Rightarrow \sum y_i - m \sum_i x_i            = c \sum 1 \\
& \Rightarrow c = \frac{\sum y_i - m \sum x_i}{\sum 1} \\
& \Rightarrow c = \frac{\sum y_i}{\sum 1} - m \frac{\sum x_i}{\sum 1} \\[0.2cm]
& \Rightarrow c = \bar{y} - m \bar{x} \\
\end{align}
$$
Solve the simultaneous equations
Here we let $n$ be the length of $x$, which is also the length of $y$.

$$
\begin{align}
& m = \frac{\sum_i x_i y_i - c \sum_i x_i}{\sum_i x_i x_i} \\[0.2cm]
& \Rightarrow m = \frac{\sum x_i y_i - (\bar{y} - m \bar{x}) \sum x_i}{\sum x_i x_i} \\
& \Rightarrow m \sum x_i x_i = \sum x_i y_i - \bar{y} \sum x_i + m \bar{x} \sum x_i \\
& \Rightarrow m \sum x_i x_i - m \bar{x} \sum x_i = \sum x_i y_i - \bar{y} \sum x_i \\[0.3cm]
& \Rightarrow m = \frac{\sum x_i y_i - \bar{y} \sum x_i}{\sum x_i x_i - \bar{x} \sum x_i} \\[0.2cm]
& \Rightarrow m = \frac{\sum (x_i y_i) - n \bar{y} \bar{x}}{\sum (x_i x_i) - n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum (x_i y_i) - n \bar{y} \bar{x} - n \bar{y} \bar{x} + n \bar{y} \bar{x}}{\sum (x_i x_i) - n \bar{x} \bar{x} - n \bar{x} \bar{x} + n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum (x_i y_i) - \sum y_i \bar{x} - \sum \bar{y} x_i + n \bar{y} \bar{x}}{\sum (x_i x_i) - \sum x_i \bar{x} - \sum \bar{x} x_i + n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum_i (x_i - \bar{x}) (y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \\
\end{align}
$$

Using sklearn neural networks

```python
import sklearn.neural_network as sknn

# Expects a 2D array of inputs.
w2d = w.reshape(-1, 1)

# Train the neural network.
regr = sknn.MLPRegressor(max_iter=10000).fit(w2d, d)

# Show the predictions.
np.array([d, regr.predict(w2d)]).T
```

```python
# The score.
regr.score(w2d, d)
```

```python

```

```python

```

CourseWork/Topic 6 Simple Linear Regression with NumPy.ipynb


```python
import numpy as np
import matplotlib.pyplot as plt
import math
import sympy as sp
import pandas as pd
import matplotlib.animation as animation
plt.style.use('seaborn-poster')

# if using a Jupyter notebook, include:
%matplotlib inline
```

```python

t, a = sp.symbols('t a')

def taylor_terms(func, order, point, derivatives=None):
    """
    find symbolic derivatives and Taylor terms
    func: symbolic function to approximate
    order: highest order derivative for Taylor polynomial (interger)
    point: point about which the function is approximated
    derivatives: list of symbolic derivatives
    """

    # initialize list of derivatives
    if derivatives is None:
        derivatives = [func.subs({t: a})]

    # check if highest order derivative is reached
    if len(derivatives) > order:
        # return list of taylor terms evaluated using substitution
        return derivatives, [derivatives[i].subs({a: point}) / math.factorial(i) * (t - point) ** i for i in range(len(derivatives))]

    # differentiate function with respect to t
    derivative = func.diff(t)
    # append to list of symbolic derivatives ** substitute t with a **
    derivatives.append(derivative.subs({t: a}))

    # recursive call to find next term in Taylor polynomial
    return taylor_terms(derivative, order, point, derivatives)

def taylor_polynomials(_terms):
    """
    find Taylor polynomials
    func: symbolic function to approximate
    order: highest order derivative for Taylor polynomial (interger)
    point: point about which the function is approximated
    derivatives: list of Taylor terms
    """

    # initialize list
    polynomials = []

    # initialize taylor polynomial
    _poly = None

    # loop through tayloer terms
    for term in range(len(_terms)):
        # build up polynomial on each iteration
        _poly = _terms[term] if _poly is None else _poly + _terms[term]

        # store current taylor polynomial
        polynomials.append(_poly)

    # return taylor polynomials
    return polynomials

if __name__ == '__main__':

    # analysis label
    label = 'ln(t)'

    # symbolic function to approximate
    f = sp.log(t)

    # point about which to approximate
    approximation_point = np.pi

    # definte time start and stop
    start = 0.01
    stop = 2 * sp.pi
    time = np.arange(start, stop, 0.1)

    # find taylor polynomial terms describing function f(t)
    symbolic_derivatives, terms = taylor_terms(func=f, order=4, point=approximation_point)
    polys = taylor_polynomials(terms)

    # initialize plot
    fig, ax = plt.subplots()
    ax.set(xlabel='t', ylabel='y', title=f'Taylor Polynomial Approximation: {label}')
    legend = []

    for p, poly in enumerate(polys):
        # plot current polynomial approximation
        ax.plot(time, [poly.subs({t: point}) for point in time])

        # append item to legend
        legend.append(f'P{p}')

    # plot actual function for comparison
    ax.plot(time, [f.subs({t: point}) for point in time])
    legend.append(f'f(t)')

    # create dataframe
    df = pd.DataFrame({'symbolic_derivatives': symbolic_derivatives,
                       'taylor_terms': terms,
                       'polynomials': polys
                       })

    # save and show results
    ax.legend(legend)
    ax.grid()
    plt.savefig(f'taylor_{label}.png')
    plt.show()

    df.to_csv(f'taylor_{label}.csv', encoding='utf-8')
    print(df.head(10))

```

```python
#new Approximation function
def func_cos(x, n):
    cos_approx = 0
    for i in range(n):
        coef = (-1)**i
        num = x**(2*i)
        denom = math.factorial(2*i)
        cos_approx += ( coef ) * ( (num)/(denom) )

    return cos_approx
angles = np.arange(-2*np.pi,2*np.pi,0.1)
p_cos = np.cos(angles)

angle_rad = (math.radians(45))
out = func_cos(angle_rad,5)
print(out)

fig, ax = plt.subplots()
plt.rcParams["figure.figsize"] = (10,10)
ax.plot(angles,p_cos)

# add lines for between 1 and 6 terms in the Taylor Series
for i in range(1,6):
    t_cos = [func_cos(angle,i) for angle in angles]
    ax.plot(angles,t_cos)

ax.set_ylim([-7,4])

# set up legend
legend_lst = ['cos() function']
for i in range(1,6):
    legend_lst.append(f'Taylor Series - {i} terms')
ax.legend(legend_lst, loc=3)

plt.show()

```


# Rectangular Plate Bending Element

This derivation follows the procedure given in Chapter 12 of "A First Course in the Finite Element Method, 4th Edition" by Daryl L. Logan.

We'll start by importing a few Python libraries that are useful for symbolic math, and initializing "pretty" printing.

```python
from sympy import symbols, Matrix, diff, integrate, simplify, factor, latex, init_printing
from IPython.display import display, Math
init_printing()
```

The plate width will be defined as $2b$, and the height will be $2c$ to be consistent with Figure 12-1. We'll set up some Sympy symbols to represent $b$ and $c$.

```python
b, c = symbols('b, c')
```

The plate is defined by four nodes specified in counter-clockwise order: i, j, m, and n. The local x-axis runs from node i toward node j, and the local y-axis runs from node i toward node n. Next we'll define the element's local displacement vector, $[d]$, at each node. There are 3 degrees of freedom at each node: $w$, $\theta_x$, and $\theta_y$.

```python
wi, theta_xi, theta_yi = symbols('w_i, theta_x_i, theta_yi')
wj, theta_xj, theta_yj = symbols('w_j, theta_xj, theta_yj')
wm, theta_xm, theta_ym = symbols('w_m, theta_xm, theta_ym')
wn, theta_xn, theta_yn = symbols('w_n, theta_xn, theta_yn')
d = Matrix([wi, theta_xi, theta_yi, wj, theta_xj, theta_yj, wm, theta_xm, theta_ym, wn, theta_xn, theta_yn])
display(Math('[d] = ' + latex(d)))
```

A 12-term polynomial displacement function will be assumed to define the out-of-plane displacement, w, at any point (x, y) in the plate's local coordinate system. The rotations about each axis are derivatives of this displacement:

$w = a_1 + a_2x + a_3y + a_4x^2 + a_5xy + a_6y^2 + a_7x^3 + a_8x^2y + a_9xy^2 + a_{10}y^3 + a_{11}x^3y + a_{12}xy^3$

$\theta_x = \frac{dw}{dy} = a_3 + a_5x + 2a_6y + a_8x^2 + 2a_9xy + 2a_{10}y^2 + a_{11}x^3 + 3a_{12}xy^2$

$\theta_y = -\frac{dw}{dx} = -a_2 - 2a_4x - a_5y - 3a_7x^2 - 2a_8xy - a_9y^2 - 3a_{11}x^2y - a_{12}y^3$

The negative sign on $\frac{dw}{dx}$ is required to be consistent with the right hand rule. These equations can be rewritten in matrix form as follows:

$[\psi] = [P][a]$

where $[\psi]$ is shorthand for  $\begin{bmatrix} w \\ \theta_x \\ \theta_y \end{bmatrix}$ and $[P]$ is defined as follows:

```python
x, y = symbols('x, y')
P = Matrix([[1, x, y, x**2, x*y, y**2, x**3, x**2*y, x*y**2, y**3, x**3*y, x*y**3],
            [0, 0, 1, 0, x, 2*y, 0, x**2, 2*x*y, 3*y**2, x**3, 3*x*y**2],
            [0, -1, 0, -2*x, -y, 0, -3*x**2, -2*x*y, -y**2, 0, -3*x**2*y, -y**3]])
display(Math('P = ' + latex(P)))
```

This equation for $[w]$ is valid for a single node. Evaluating $[P]$ at each node gives us a larger set of equations:

$[d] = [C][a]$

where $[C]$ is merely $[P]$ evaluated at each node, and $[d]$ is correpsondingly $[\psi]$ at each node. Knowing that the plate width is $2b$ and the plate height is $2c$, we can obtain the matrix $[C]$.

```python
C = Matrix([P, P, P, P])
C[0:3, 0:12] = C[0:3, 0:12].subs(x, 0).subs(y, 0)      # i-node @ x = 0, y = 0
C[3:6, 0:12] = C[3:6, 0:12].subs(x, 2*b).subs(y, 0)    # j-node @ x = 2b, y = 0
C[6:9, 0:12] = C[6:9, 0:12].subs(x, 2*b).subs(y, 2*c)  # m-node @ x = 2b, y = 2c
C[9:12, 0:12] = C[9:12, 0:12].subs(x, 0).subs(y, 2*c)  # n-node @ x = 0, y = 2c
display(Math('[C] = ' + latex(C)))
```

An important matrix that we will come back to later is the shape function matrix $[N]$, defined as:

$[N] = [P][C]^{-1}$

The closed form solution of $[N]$ for a rectangular plate is:

```python
N = P*C.inv()
display(Math('[N] = ' + latex(simplify(N))))
```

We can now solve for the $[a]$ matrix in terms of the nodal displacements:

```python
a = simplify(C.inv()*d)
display(Math('[a] = ' + latex(a)))
```

The next step is to define the curvature matrix:

$[\kappa] = \begin{bmatrix} -\frac{d^2w}{dx^2} \\ -\frac{d^2w}{dy^2} \\ -\frac{2d^2w}{dxdy} \end{bmatrix} = [Q][a]$

It should be recognized that $w/[a]$ is simply the first row of our $[P]$ matrix. Evaluating the derivatives in this expression gives $[Q]$ as follows:

```python
Q = Matrix([-diff(diff(P[0, :], x), x),
            -diff(diff(P[0, :], y), y),
            -2*diff(diff(P[0, :], x), y)])
display(Math('[Q] = ' + latex(Q)))
```

With $[Q]$ in hand we can now solve for the $[B]$ matrix which is essential for formulating the stiffness matrix $[k]$

```python
B = simplify(Q*C.inv())
display(Math('[B] = ' + latex(B)))
```

Now we form the constitutive matrix for isotropic materials, [D]. This matrix is analagous to the flexural stiffness of a beam EI.

```python
E, t, nu = symbols('E, t, nu')
Coef = E*t**3/(12*(1-nu**2))
D = Coef*Matrix([[1, nu, 0],
                 [nu, 1, 0],
                 [0, 0, (1-nu)/2]])
display(Math('[D] = ' + latex(D)))
```

Now we can calculate the stiffness matrix:

$[k] = \int_0^{2c} \int_0^{2b} [B]^T[D][B] dx dy$

```python
k = integrate(integrate(B.T*D*B, (x, 0, 2*b)), (y, 0, 2*c))
display(Math('[k] = {Et^3}/{12(1-\nu^2)}' + latex(simplify(k/Coef))))
```

The surface force matrix $[F_s]$ can be obtained from the shape function matrix. Since we're interested in the surface force matrix for uniform pressures in the direction of w,

```python
q = symbols('q')
Fs = integrate(integrate(N[0, :].T*q, (x, 0, 2*b)), (y, 0, 2*c))
display(Math('[F_s] = 4qcb' + latex(Fs/(4*q*c*b))))
print(Fs/(4*q*c*b))
```

```python

```


## Setting up python environment

```python
# reset all previously defined varibles
%reset -f

# import everything from sympy moduleb
from sympy import *

# pretty math formatting
init_printing()   # latex
```

### Symbolic variables must be declared

```python
x,y,z = symbols('x y z')
```

###  Function definition

```python
## Example 1

f = 2*x**3  # define function
f           # print function

```

```python
## Example 2

g = x**2 + x*y**2 # define function
g           # print function
```

### Function Evaluation

```python
# f(0)
f.subs(x,0)
```

```python
# f(1)
f.subs(x,1)
```

```python
# g(x=2,y=3)
g.subs([(x,2),(y,3)])
```

## Plotting

```python
## Graph of f = x^2

f = x**2

plot(f)
```

```python
## multiple graph in same window

f = x**2
g = x**3

p = plot(f,g, (x, -2, 2),show=False, legend=True  )

p[0].line_color = 'blue'
p[1].line_color = 'green'

p[0].label      = 'f(x)'
p[1].label      = 'g(x)'
p.show()

```

```python
## undocking plots out of browser
%matplotlib qt

f = x**2
plot(f)

```

    <sympy.plotting.plot.Plot at 0x7f5b173ce850>

```python
## forcing plots inside browser
%matplotlib inline

f = x**2
plot(f)
```

```python

```


## Exercise 1 (3 points): Principal Component Analysis [Python]

In this exercise, we cover probably one of the most popular data analysis techniques: principal component analysis (PCA). It is used in many applications and is often part of the pre-processing steps. With its help, we can find new axes which are designed to reflect the variation present in the data. What is more, we can represent each data point relative to these new axes and when we choose fewer axes we effectively project our data to a lower-dimensional projection space. This way, PCA can also be used as a dimensionality reduction technique where we keep only the dimensions which explain most of the variance. However, this specific aspect is postponed until Exercise 3.

There are far too many applications of PCA to be covered in a single exercise and you will probably encounter some of them in other lectures anyway. However, we still want to look at an example to understand the basics.

Suppose you have captured an image of a brick as shown in Figure 1 (a). Additionally, you have already extracted a mask which discriminates the background from the brick. The file shapeData.pickle is attached to this exercise and contains a matrix of size 6752 × 2 which stores a list of all non-zero positions of this mask. We are going to apply PCA to this data and will see what we can do with it. For the test script, please pay also attention to Table 1.

1. Load the data points stored in the file shapeData.pickle and plot the result. Hint: set an equal axes scaling so that the mask of the brick does not look distorted

2. Centre the data so that it has zero mean, i.e. when $X \in \mathbb{R}^{nx2}$ denotes our data matrix, calculate
\begin{equation} X =  \tilde{X} - (\mu_1 \hspace{0.3cm} \mu_2)\end{equation}
where $\mu_i$ denotes the mean of the 𝑖-th column of $\tilde{X}$

3. Compute the covariance matrix
\begin{equation}C = \frac{1}{n-1}X^TX \end{equation}

4. We can now calculate the eigensystem, i.e. the principal components.

    a) Compute the eigenvalues and corresponding eigenvectors of $C$. You can use the $np.linalg.eig$ function for this task. But make sure (like in one of the previous exercises) that the result is sorted descendingly by the eigenvalues.

    b) The eigenvalue $\lambda_i$ corresponds to the variance along the new axis $u_i$ (which is defined by the respective eigenvector). Calculate the explained variance of each axis (in percentage terms).

5. The eigensystem $U = (u_1 \hspace{0.3cm} u_2)$ defines its own feature space and we can describe our points relative to the new axes. Project each data point to the new eigensystem and plot the result. Before you do so, think about what you would expect the shape to change to and then check if the result matches with your expectation.

## Exercise 2 (4 points): Singular Value Decomposition

Singular value decomposition (SVD) is related to PCA since it ends up with a similar result even though the computation is different. In this exercise, we want to discuss the similarities and the differences between these two approaches.

Roughly speaking, SVD is an extended diagonalization technique which allows us to decompose our data matrix
\begin{equation}X = U\Sigma V^T\end{equation}

As usual, $X \in \mathbb{R}^{n×d}$ stores the observations in rows and the variables in columns. Here, we assume that $X$ is already centred, i.e. the columns have zero mean. $\Sigma$ is a diagonal matrix storing the singular values $s_i$ and the column vectors of $U$ and $V$ contain the left and right singular vectors, respectively.

In PCA, we need to calculate the covariance matrix $C$ (Equation 2). This matrix is symmetric so we can diagonalize it directly
\begin{equation} C = \frac{1}{n-1}X^TX = WDW^T \end{equation}
where 𝑊 stores the eigenvectors in the columns and 𝐷 has the eigenvalues 𝜆𝑖 on the diagonal.

1. [Pen and Paper] Let’s begin with the matrices of SVD and see how they are related to PCA. For this, fill out the missing properties summarized in Table 2.

    a) For the matrices $U$ and $V$, do the singular vectors correspond to the eigenvectors of $X^TX$ or $XX^T$? For this, show the mappings of the third column of Table 2 with the help of Equation 3, Equation 4 and \begin{equation} \tilde{C} = \frac{1}{n-1}X^TX = \tilde{W}\tilde{D}\tilde{W}^T \end{equation}

    b) Based on your previous findings, what is the relationship between the singular values $s_i$ and the eigenvalues $\lambda_i$? \begin{equation} s_i = … \end{equation}

    c) Decide about the matrix dimensions (the mappings should help with this decision). Assume that zero rows/columns have not been removed.

    d) We can also interpret the matrices geometrically. What is the corresponding affine transformation (translation, scaling, rotation, etc.) of each matrix? Hint: think about the special properties of the matrices

2. [Python] We saw that there is a strong relationship between the result of PCA and SVD since we can define a mapping between the eigenvectors/eigenvalues and singular vectors/singular values. However, the algorithm used to compute the results is different. For the standard approach with PCA1, we need to calculate the covariance matrix but this is not the case in SVD. It turns out that this is an advantage for SVD since these calculations can get numerically unstable. We now want to explore this circumstance a bit further by calculating the singular values of the matrix

\begin{equation} \begin{pmatrix}1 & 1 & 1\\ \epsilon & 0 & 0\\ 0 & \epsilon & 0\\ 0 & 0 & \epsilon \end{pmatrix} \end{equation}

 where $\epsilon \in \mathbb{R}^+$ is a small number. We are not interested in the corresponding vectors in this part.

    a) Define a function which takes 𝜀 as an argument and defines the matrix 𝐴 accordingly.
        i. Calculate the singular values via the standard PCA method
\begin{equation} C = A^TA \\ \lambda = eval(C) \\ s = \sqrt{\lambda} \end{equation}

        It is helpful to calculate this in a step-by-step fashion and to print the exact definition after each step with full precision. For the latter, you can use the astype(str) method of the ndarray objects.

        ii. Calculate the singular values $s$ again but this time using an SVD algorithm (e.g.np.linalg.svd) and print the result.

    b) Call your function and set $\epsilon$ to the tiny number $10^{-7}$
\begin{equation} epsilon = 1.0e-7 \end{equation}

    c) Now, set 𝜀 = 10−8 and repeat the calculations. If done correctly, you should get an error. What is the problem here?

    d) How can you tell that the standard PCA approach returns at least one incorrect result for the eigenvalues $\lambda$? Hint: look for a property of the covariance matrix which is violated here.

## Exercise 3 (3 points): PCA vs. LDA [Python]

As mentioned in Exercise 1, it is also common to use PCA (or SVD) as a dimensionality reduction technique. Instead of projecting our data points to all new axes, we select only a subset and this subset is designed to cover as much of the variance in the data as possible.

We already saw another dimensionality reduction technique in previous assignments, namely
linear discriminant analysis (LDA). We are going to apply PCA also to the Statlog (Vehicle
Silhouettes) dataset and compare the result with the one obtained by LDA. For the test script,
please pay also attention to Table 3.
1. Load the data from the train.csv file into your Python program.

2. PCA may not work well when the features are scaled differently as it is the case with our
dataset. As a countermeasure, standardize the data (as discussed in assignment 3). You
can use the StandardScaler from the sklearn package for this purpose.

3. Apply PCA to the dataset and calculate the (two-dimensional) projections of the data
to the new coordinate system. There is a PCA class in the sklearn.decomposition
package which you can use.

4. Show the explained variance (in percentage terms) as a cumulative sum over all eigenvalues
in a plot (Figure 2 shows an example solution).

5. Plot the data with respect to the first two principal components (𝒖1 and 𝒖2) and compare
the result with your LDA plot.

6. Fill out Table 4 which compares some properties of PCA with LDA.

    a) Is the procedure supervised or unsupervised?

    b) What is the focus of the new axes, i.e. are they designed to explain the variance in
the data or to discriminate the classes?

    c) What is the maximal number of eigenvectors we can retrieve, i.e. the maximal dimension size of the projection space?


# Praca domowa - Dominik Stańczak

```python
import sympy
sympy.init_printing()
t, lambda3a, lambdaa12, N4, N12, N16, dN4, dN12, dN16, dt = sympy.symbols('t, lambda_3a, lambda_a12, N4, N12, N16, dN4, dN12, dN16, dt', real=True)

eqs = [
    sympy.Eq(dN4/dt, -3*lambda3a * N4 **3 - lambdaa12 * N4 * N12),
    sympy.Eq(dN12/dt, lambda3a * N4 **3 - lambdaa12 * N4 * N12),
    sympy.Eq(dN16/dt, lambdaa12 * N4 * N12)
]
eqs
```

```python
m, rho = sympy.symbols('m, rho', real=True)
X4, X12, X16, dX4, dX12, dX16 = sympy.symbols('X4, X12, X16, dX4, dX12, dX16', real=True)
Xeqs = [
    sympy.Eq(X4, m/rho*4*N4),
    sympy.Eq(X12, m/rho*12*N12),
    sympy.Eq(X16, m/rho*16*N16),
]
Xeqs
```

```python
subs = {X4: dX4, X12: dX12, X16: dX16, N4: dN4, N12: dN12, N16: dN16}
dXeqs = [eq.subs(subs) for eq in Xeqs]
dXeqs
```

```python
full_conservation = [sympy.Eq(X4 + X12 + X16, 1), sympy.Eq(dX4 + dX12 + dX16, 0)]
full_conservation
```

```python
all_eqs = eqs + Xeqs + dXeqs + full_conservation
all_eqs
```

```python
X_all_eqs = [eq.subs(sympy.solve(Xeqs, [N4, N12, N16])).subs(sympy.solve(dXeqs, [dN4, dN12, dN16])) for eq in eqs] + [full_conservation[1]]
X_all_eqs
```

```python
solutions = sympy.solve(X_all_eqs, [dX4, dX12, dX16])
dX12dX4 = solutions[dX12]/solutions[dX4]
dX12dX4
```

```python
q = sympy.symbols('q', real=True)
dX12dX4_final = dX12dX4.subs({lambdaa12*m: q * lambda3a * rho}).simplify()
dX12dX4_final
```

```python
fX12 = sympy.Function('X12')(X4)
diffeq = sympy.Eq(fX12.diff(X4), dX12dX4_final.subs(X12, fX12))
diffeq
```

```python
dX16dX4 = solutions[dX16]/solutions[dX4]
dX16dX4
```

```python
dX16dX4_final = dX16dX4.subs({lambdaa12*m: q * lambda3a * rho}).simplify()
dX16dX4_final
```

```python
derivatives_func = sympy.lambdify((X4, X12, X16, q), [dX12dX4_final, dX16dX4_final])
derivatives_func(1, 0, 0, 1)
```

```python
def f(X, X4, q):
    return derivatives_func(X4, *X, q)
f([0, 0], 1, 1)
```

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt
X4 = np.linspace(1, 0, 1000)
q_list = np.logspace(-3, np.log10(2), 500)
results = []
# fig, (ax1, ax2) = plt.subplots(2, sharex=True, figsize=(10, 8))
# ax1.set_xlim(0, 1)
# ax2.set_xlim(0, 1)
# ax1.set_ylim(0, 1)
# ax2.set_ylim(0, 1)
for q in q_list:
    X = odeint(f, [0, 0], X4, args=(q,))
    X12, X16 = X.T
#     ax1.plot(X4, X12, label=f"q: {q:.1f}")
#     ax2.plot(X4, X16, label=f"q: {q:.1f}")
#     ax2.set_xlabel("X4")
#     ax1.set_ylabel("X12")
#     ax2.set_ylabel("X16")
#     plt.plot(X4, X16)
#     plt.legend()
    results.append(X[-1])
results = np.array(results)
```

```python
X12, X16 = results.T
plt.figure(figsize=(10, 10))
plt.plot(q_list, X12, label="X12")
plt.plot(q_list, X16, label="X16")
plt.xlabel("q")
plt.xscale("log")
plt.ylabel("X")
plt.legend(loc='best')
plt.xlim(q_list.min(), q_list.max());
plt.grid()
plt.savefig("Reacts.png")
```


Asset price under $\mathbb Q$ follows
$$S_{t} = S_{0} exp \{ \mu t + \sigma W_{t} \}$$
Coonsider Digital put with payoff
$$h(S_{T}) = I(S_{T} < S_{0}e^{-b})$$
We want to find the forward price:
$$v = \mathbb E^{\mathbb Q}[h(S_{T})]$$
Parameters are given as
$$r = .03, \sigma = .2, \mu = r - \frac{1}{2} \sigma^2 = .01, T = 1, b = .39$$

- Prove that the exact price is 0.02275

__Pf:__

As we have discussed in the video, we claim that

\begin{equation}
\begin{aligned}
v &= \mathbb E^{\mathbb Q}[h(S_{T})] \\
&= \mathbb Q(W_1 < - \frac{b + \mu}{\sigma} = -2) \\
&= \Phi(-2), \\
\end{aligned}
\end{equation}

where $W_1 \sim \mathcal N(0,1)$ and $\Phi(\cdot)$ is the $\mathrm{CDF}$ of the standard Normal Distribution.

Thus, the exace price is shown as follows:

```
import scipy.stats as ss
exact_price = ss.norm(0,1).cdf(-2)
print(exact_price)
```

    0.022750131948179195

- Use $\mathrm{OMC}$ find the price

__Soln:__

By using $\mathrm{OMC}$, we replace $\Phi(-2)$ with $\frac{1}{n} \sum_{i=1}^n I(X_i<-2)$

```
import numpy as np

def OMC(N):         # N is the number of samples
  s = 0
  for i in range(N):
    X = np.random.normal(0,1)
    if (X<-2):
      s += 1
  return s/N
OMC(1000000)
```

    0.022768

- Use $\mathrm{IS}(\alpha)$ find the price

 __Soln:__

 Assume $\phi_\alpha(\cdot) \sim \mathcal N(-\alpha,1)$, where $\phi_\alpha(\cdot)$ is the $\mathrm{pdf}$.

Then we can rewrite v which has been discussed in the video:

\begin{equation}
\begin{aligned}
v &= \mathbb E[I(Y<-2) \cdot e^{\frac{1}{2}\alpha^2 + \alpha Y}|Y \sim \phi_\alpha] \\
&\approx \frac{1}{n} \sum_{i=1}^n I(Y_i<-2) \cdot e^{\frac{1}{2}\alpha^2 + \alpha Y_i}, \\
\end{aligned}
\end{equation}

where $Y_i \sim \mathcal N(-\alpha,1).$

```
import numpy as np
import math

def ImpSamp(N,alpha):         # N is the number of samples
  s = 0
  for i in range(N):
    Y = np.random.normal(-alpha,1)
    if (Y<-2):
      s += math.e ** (alpha**2/2 + alpha*Y)
  return s/N
ImpSamp(10000,2)
```

    0.02249178731207442

- Can you show your approach is optimal?

```
err_list = []
for alpha in range(10):
  M = 1000                 # the number of trials to find the minimum of error
  error = []
  for i in range (M):
    error.append(abs(ImpSamp(10000,alpha) - exact_price))     # document every trial's error
  err_list.append(min(error))       # document the minimum of error wrt different alpha
min_err = min(err_list)
print(">>> The minimum error among different α is " + str(min_err) + ", where α = " + str(err_list.index(min_err)))
```

    >>> The minimum error among different α is 2.0011788371201988e-07, where α = 2

- Prove or demonstrate $\mathrm{IS}$ is more efficient to $\mathrm{OMC}$

```
N = 1000                         # N is the number of samples for both OMC and Importance Sampling
sq_err_omc = 0
sq_err_ImpSamp = 0
K = 10000                         # K is the number of trials for OMC
for i in range(K):
  sq_err_omc += (exact_price - OMC(N))**2
  sq_err_ImpSamp += (exact_price - ImpSamp(N,2))**2
mse_omc = sq_err_omc / K
mse_ImpSamp = sq_err_ImpSamp / K
print(">>> The Mean Square Error of Ordinary Monte Carlo is " + str(mse_omc))
print(">>> The Mean Square Error of Importance Sampling is " + str(mse_ImpSamp))
```

    >>> The Mean Square Error of Ordinary Monte Carlo is 2.228314286581901e-05
    >>> The Mean Square Error of Importance Sampling is 1.1975332166123286e-06

- After trying two different values of $K$ ($K=1000\ or\ 10000$), the value of exponential will not change. Thus we can say that $\mathrm{IS}$ is more efficient to $\mathrm{OMC}$.

__Numeric Method:__

As we have shown before,
$$ \hat v_{OMC} = \frac{1}{n} \sum_{i=1}^n I(X_i<-2),$$
and
$$ \hat v_{IS} = \frac{1}{n} \sum_{i=1}^n I(Y_i<-2) \cdot e^{\frac{1}{2}\alpha^2 + \alpha Y_i}.$$
Both $v_{OMC}$ and $v_{IS}$ are unbiased, thus we can deduce that

\begin{equation}
\begin{aligned}
\mathrm{MSE}(\hat v_{OMC}) &= Var(\hat v_{OMC}) \\
&= \frac{1}{n} \left( \mathbb E[I(X_i<-2)^2] - (\mathbb E[I(X_i<-2)])^2 \right) \\
&= \frac{1}{n}(\Phi(-2) - \Phi(-2)^2) \\
\mathrm{MSE}(\hat v_{IS}) &=  Var(\hat v_{IS}) \\
&= \frac{1}{n} \left( \mathbb{E} [I(Y_{i} < - 2) e^{\alpha^{2} + 2 \alpha Y_{i}}] - \Phi^{2}(-2) \right) \\
\end{aligned}
\end{equation}

Now we calculate the right hand side of the last equation:

\begin{equation}
\begin{aligned}
\mathbb{E} [I(Y_{i} < - 2) e^{\alpha^{2} + 2 \alpha Y_{i}}] &= \int_{- \infty}^{-2} e^{\alpha^{2}+ 2 \alpha y}  \frac{1}{\sqrt{2 \pi}} e^{- \frac{(y + \alpha)^2}{2}} \mathrm{d} y \\
&= \int_{- \infty}^{-2} \frac{1}{\sqrt{2 \pi}} e^{- \frac{y^{2} - \alpha y - \alpha^{2}}{2}} \mathrm{d} y \\
&= \int_{- \infty}^{-2} \frac{1}{\sqrt{2 \pi}} e^{- \frac{(y - \alpha)^{2}}{2}}  e^{\alpha^{2}} \mathrm{d} y  \\
&= e^{\alpha^{2}} \cdot \Phi(-2-\alpha), \\
\end{aligned}
\end{equation}

Thus, we will have
$$\mathrm{MSE}(\hat v_{IS}) = \frac{1}{n} \left( e^{\alpha^{2}} \Phi(-2-\alpha) - \Phi^{2}(-2) \right).$$

Therefore,
$$\mathrm{MSE}(\hat v_{OMC}) - \mathrm{MSE}(\hat v_{IS}) = \frac{1}{n} \left( \Phi(-2) - e^{\alpha^{2}} \Phi(-2 - \alpha) \right) \ge 0.$$

```
import numpy as np
import scipy.stats as ss

for alpha in range(5):
  diff = ss.norm(0,1).cdf(-2) - np.exp(alpha**2) * ss.norm.cdf(-2-alpha)
  print(diff)
```

    0.0
    0.019080728658526478
    0.021020940734838522
    0.020427370203270685
    0.01398320509620665


```python
import math
import torch
import numpy as np
import matplotlib.pyplot as plt
```

```python
device = 'cuda' if torch.cuda.is_available() else 'cpu'
```

# Exploring optimisation of analytic functions

```python
def rastrigin(X, A=1.0):
    return A*2 + ( (X[0]**2 - A*torch.cos(2*math.pi*X[0])) + (X[1]**2 - A*torch.cos(2*math.pi*X[1])) )
```

```python
xmin, xmax, xstep = -5, 5, .2
ymin, ymax, ystep = -5, 5, .2
xs = np.arange(xmin, (xmax + xstep), xstep)
ys = np.arange(ymin, (ymax + ystep), ystep)
z = rastrigin(torch.tensor([xs,ys]), A=1.0).numpy()
```

```python
fig, ax = plt.subplots(figsize=(8,6))
ax.plot(xs, z)
plt.tight_layout()
plt.savefig('rastrigin.png')
```

With $A=1.0$ we find the Rastrigin function has many small 'bumps' or local minima. However the main basin at $x=0$ is the global minimum we want our optimisers to reach.

```python
p_SGD = torch.tensor([5.0, 5.0], requires_grad=True, device=device)
p_SGD_Mom = torch.tensor([5.0, 5.0], requires_grad=True, device=device)
p_Adagrad = torch.tensor([5.0, 5.0], requires_grad=True, device=device)
p_Adam = torch.tensor([5.0, 5.0], requires_grad=True, device=device)
print('Parameters initialised:\n  ', p_SGD, p_SGD.type())
epochs = 100
print('Max epochs:\n  ', epochs)
A = 1.0
opt_SGD = torch.optim.SGD([p_SGD], lr=0.01)
print(f'Initialised SGD:\n  Learning rate:{0.01}')
opt_SGD_Mom = torch.optim.SGD([p_SGD_Mom], lr=0.01, momentum=0.09)
print(f'Initialised SGD Momentum:\n  Learning rate:{0.01}, Momentum:{0.09}')
opt_Adagrad = torch.optim.Adagrad([p_Adagrad], lr=0.01)
print(f'Initialised Adagrad:\n  Learning rate:{0.01}')
opt_Adam = torch.optim.Adam([p_Adam], lr=0.01)
print(f'Initialised Adam:\n  Learning rate:{0.01}')

plt_loss_SGD = []
plt_loss_SGD_Mom = []
plt_loss_Adagrad = []
plt_loss_Adam = []

for epoch in range(epochs):
    # zero gradients
    opt_SGD.zero_grad()
    opt_SGD_Mom.zero_grad()
    opt_Adagrad.zero_grad()
    opt_Adam.zero_grad()
    # compute loss
    loss_SGD = rastrigin(p_SGD, A=A)
    loss_SGD_Mom = rastrigin(p_SGD_Mom, A=A)
    loss_Adagrad = rastrigin(p_Adagrad, A=A)
    loss_Adam = rastrigin(p_Adam, A=A)
    # backprop
    loss_SGD.backward()
    loss_SGD_Mom.backward()
    loss_Adagrad.backward()
    loss_Adam.backward()
    # step optimiser
    opt_SGD.step()
    opt_SGD_Mom.step()
    opt_Adagrad.step()
    opt_Adam.step()
    # store loss for plots
    plt_loss_SGD.append(loss_SGD.item())
    plt_loss_SGD_Mom.append(loss_SGD_Mom.item())
    plt_loss_Adagrad.append(loss_Adagrad.item())
    plt_loss_Adam.append(loss_Adam.item())

print(f'Loss function:\n  Rastrigin, A={A}')
fig, ax = plt.subplots(figsize=(8,5))
ax.plot(plt_loss_SGD, label='SGD', linewidth=2, alpha=.6)
ax.plot(plt_loss_SGD_Mom, label='SGD Momentum', linewidth=2, alpha=.6)
ax.plot(plt_loss_Adagrad, label='Adagrad', linewidth=2, alpha=.6)
ax.plot(plt_loss_Adam, label='Adam', linewidth=2, alpha=.6)

ax.legend()
ax.set_xlabel('Epoch')
ax.set_ylabel('Loss')
plt.tight_layout()
plt.savefig('optimiser_comparison_rastrigin.png')
```

Rastrigin is a difficult function to optimise as it's filled with many local minima, but there is only one global minimum. We find that SGD + Momentum shows the best performance when applied to a 2D Rastrigin function with $A=1.0$ (a parameter which determines how 'bumpy' the function is).

# Optimisation of a SVM on real data

Applying soft-margin SVM to Iris data and optimise its paramters using gradient descent.

Note: we will only be using two of the four classes from the dataset.

An SVM tries to find the maximum margin hyperplane which separates the data classes. For a soft margin SVM
where $\textbf{x}$ is our data, we minimize:

\begin{equation}
\left[\frac 1 n \sum_{i=1}^n \max\left(0, 1 - y_i(\textbf{w}\cdot \textbf{x}_i - b)\right) \right] + \lambda\lVert \textbf{w} \rVert^2
\end{equation}

We can formulate this as an optimization over our weights $\textbf{w}$ and bias $b$, where we minimize the
hinge loss subject to a level 2 weight decay term. The hinge loss for some model outputs
$z = \textbf{w}\textbf{x} + b$ with targets $y$ is given by:

\begin{equation}
\ell(y,z) = \max\left(0, 1 - yz \right)
\end{equation}

```python
import torch
import pandas as pd
from tqdm.auto import tqdm
import matplotlib.pyplot as plt
```

```python
def svm(x,w,b):
    h = (w*x).sum(1) + b
    return h
```

```python
def hinge_loss(z, y):
    yz = y * z
    return torch.max(torch.zeros_like(yz), (1-yz))
```

```python
# test
print(hinge_loss(torch.randn(2), torch.randn(2) > 0).float())
```

    tensor([1.0000, 2.7751])

```python
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'
df = pd.read_csv(url, header=None)
df = df.sample(frac=1, random_state=0) # shuffle

df = df[df[4].isin(['Iris-virginica', 'Iris-versicolor'])] # filter

# add label indices column
mapping = {k: v for v, k in enumerate(df[4].unique())}
df[5] = (2 * df[4].map(mapping)) - 1 # labels in {-1,1}

# normalise data
alldata = torch.tensor(df.iloc[:, [0,1,2,3]].values, dtype=torch.float)
alldata = (alldata - alldata.mean(dim=0)) / alldata.var(dim=0)

# create datasets
targets_tr = torch.tensor(df.iloc[:75, 5].values, dtype=torch.long)
targets_va = torch.tensor(df.iloc[75:, 5].values, dtype=torch.long)
data_tr = alldata[:75]
data_va = alldata[75:]
```

```python
from torch.utils import data
```

```python
# mini-batch training data
dataset_tr = data.TensorDataset(data_tr, targets_tr)
dataloader_tr = data.DataLoader(dataset_tr, batch_size=25, shuffle=True)
```

```python
# mini-batch test data
dataset_va = data.TensorDataset(data_va, targets_va)
dataloader_va = data.DataLoader(dataset_va, batch_size=25, shuffle=True)
```

```python
def eval_accuracy(predictions, labels):
    a = sum(predictions.detach().numpy() * labels.numpy() >=0) / len(labels)
    return a
```

```python
def svm_train(train_dataloader, data_va, targets_va, epochs=100, lr=0.01, decay=0.01):
    print('Support Vector Machine:')
    print('  learning_rate:', lr)
    print('  epochs:', epochs)
    train_rows, train_col = train_dataloader.dataset.tensors[0].shape
    print(f'  X_train.shape: ({train_rows},{train_col})')
#     train_y_rows = dataloader.dataset.tensors[1].shape[0]
#     print(f'  y_train.shape: ({train_y_rows})')
#     test_rows, test_col = test_dataloader.dataset.tensors[0].shape
#     print(f'  X_test.shape: ({train_rows},{train_col})')
#     test_y_rows = dataloader.dataset.tensors[1].shape[0]
#     print(f'  y_test.shape: ({test_y_rows})')
    print('---------------------------------')
    # initialise weights and biases
    w = torch.randn(1, train_col, requires_grad=True)
    b = torch.randn(1, requires_grad=True)

    optimiser = torch.optim.SGD([w,b], lr=lr, weight_decay=decay)
#     optimiser = torch.optim.Adam([w,b], lr=lr, weight_decay=decay)

    # record loss over epoch
    losses = []
    # record training accuracy over epoch
    train_accuracy = []
    # record validation accuracy over epoch
    test_accuracy = []

    print('Training model, please wait...')
    for epoch in tqdm(range(epochs)):
        ep_loss = 0
        for train_batch in train_dataloader:
            # zero gradients
            optimiser.zero_grad()
            # compute loss
            X_train, y_train = train_batch
            y_pred = svm(X_train, w, b)
            loss = hinge_loss(y_pred, y_train).mean()
            # backprop
            loss.backward()
            # step optimiser
            optimiser.step()
            # track loss
            ep_loss += loss.item()
        losses.append(ep_loss)
        # training accuracy
        ep_train_pred = svm(X_train, w, b)
        ep_train_acc = eval_accuracy(ep_train_pred, y_train)
        train_accuracy.append(ep_train_acc)
        #   validation accuracy
        ep_test_pred = svm(data_va, w, b)
        ep_test_acc = eval_accuracy(ep_test_pred, targets_va)
        test_accuracy.append(ep_test_acc)

    print(f'Training accuracy: {train_accuracy[-1]*100}%')
    print(f'Validation accuracy: {ep_test_acc*100}%')

    print(f'Train loss: {losses[-1]}')
    return train_accuracy, test_accuracy, losses
```

```python
tr_acc, va_acc, loss = svm_train(dataloader_tr, data_va, targets_va,
                                 epochs=100, lr=0.01, decay=0.01)
```

    Support Vector Machine:
      learning_rate: 0.01
      epochs: 100
      X_train.shape: (75,4)
    ---------------------------------
    Training model, please wait...

      0%|          | 0/100 [00:00<?, ?it/s]

    Training accuracy: 92.0%
    Validation accuracy: 92.0%
    Train loss: 0.40005533397197723

```python
# SGD training accuracy
fig, ax = plt.subplots(figsize=(6,3))
ax.plot(range(len(tr_acc)), tr_acc, label='Training Accuracy (SGD)')
ax.plot(range(len(va_acc)), va_acc, label='Validation Accuracy (SGD)')
ax.set_xlabel('Epoch')
ax.set_ylabel('Accuracy')
ax.grid(True)
ax.legend(loc='best', frameon=True)
plt.tight_layout()
plt.savefig('sgd_tr_va_acc.png')
```

```python
# Adam training accuracy
fig, ax = plt.subplots(figsize=(6,3))
ax.plot(range(len(tr_acc)), tr_acc, label='Training Accuracy (Adam)')
ax.plot(range(len(va_acc)), va_acc, label='Validation Accuracy (Adam)')
ax.set_xlabel('Epoch')
ax.set_ylabel('Accuracy')
ax.grid(True)
ax.legend(loc='best', frameon=True)
plt.tight_layout()
plt.savefig('adam_tr_va_acc.png')
```

```python
tr_acc_SGD, va_acc_SGD, loss_SGD = svm_train(dataloader_tr, data_va, targets_va,
                                 epochs=100, lr=0.01, decay=0.01)
```

    Support Vector Machine:
      learning_rate: 0.01
      epochs: 100
      X_train.shape: (75,4)
    ---------------------------------
    Training model, please wait...

      0%|          | 0/100 [00:00<?, ?it/s]

    Training accuracy: 96.0%
    Validation accuracy: 88.0%
    Train loss: 0.5745970010757446

```python
tr_acc_Adam, va_acc_Adam, loss_Adam = svm_train(dataloader_tr, data_va, targets_va,
                                 epochs=100, lr=0.01, decay=0.01)
```

    Support Vector Machine:
      learning_rate: 0.01
      epochs: 100
      X_train.shape: (75,4)
    ---------------------------------
    Training model, please wait...

      0%|          | 0/100 [00:00<?, ?it/s]

    Training accuracy: 96.0%
    Validation accuracy: 88.0%
    Train loss: 0.2739289551973343

```python
# SGD vs Adam validation accuracy
fig, ax = plt.subplots(figsize=(6,3))
ax.plot(range(len(va_acc)), va_acc_SGD, label='Validation Accuracy (SGD)')
ax.plot(range(len(va_acc)), va_acc_Adam, label='Validation Accuracy (Adam)')
ax.set_xlabel('Epoch')
ax.set_ylabel('Accuracy')
ax.grid(True)
ax.legend(loc='best', frameon=True)
plt.tight_layout()
plt.savefig('sgd_vs_adam_acc_1.png')
```

SGD makes bigger jumps in accuracy whilst Adam makes smaller ones. This results in SGD obtaining higher final validation accuracy.

```python
fig, ax = plt.subplots(figsize=(6, 3))
ax.plot(range(len(loss)), loss_SGD, label='Training Loss (SGD)')
ax.plot(range(len(loss)), loss_Adam, label='Training Loss (Adam)')
ax.set_xlabel('Epoch')
ax.set_ylabel('Loss')
ax.grid(True)
ax.legend(loc='best', frameon=True)
plt.tight_layout()
plt.savefig('sgd_vs_adam_loss.png')
```

Adam's loss starts off at a very large value and it does a good job of minimising it such that it matches the superior performance of SGD.


# Examples on Using PyModPDE
---
__Mokbel Karam, James C. Sutherland and Tony Saad__

__Department of Chemical Engineering, University of Utah__

Below are the necessary libraries used in the examples below.

```python
import matplotlib.pyplot as plt
plt.rc("font", family='serif')
import numpy as np
from matplotlib import cm, animation
```

##  Example Using FTUS on Advection PDE (1D)
---

Consider the following PDE
\begin{equation}
u_t=-a u_x ; a>0
\end{equation}
to be solved using forward Euler discretization in time and upwind discretization in space(FTUS). The difference equation is
\begin{equation}
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=-a\left(\frac{u_{i}^{n}-u_{i-1}^{n}}{\Delta x}\right).
\end{equation}

```python
from pymodpde import DifferentialEquation, symbols, i, k, n # import the differential equation and the indicies we need to use

a= symbols('a') # define positive a constant for the advection velocity

DE1 = DifferentialEquation(dependentVarName='u',independentVarsNames=['x']) # construct a one D differential equation with 'x' and 'u' as independent and dependent variables, respectively

# method I of constructing the rhs:
# construct the first upwind derivative of the dependent variabe 'u'
# with respect to the independent variable 'x' using the stencil -1, and 0 evaluated at time tn = n
advectionTerm1 = DE1.expr(order=1, directionName='x', time=n, stencil=[-1, 0])

# set the rhs of the differential equation
DE1.set_rhs(- a * advectionTerm1 )

# compute the modified equation up to two terms

DE1.generate_modified_equation(nterms=4)
```

```python
# display the mofified equation
DE1.display_modified_equation()
```

$\displaystyle \frac{\partial u }{\partial t}  =  - a \frac{\partial u }{\partial x}  + \frac{a \left(- \Delta{t} a + \Delta{x}\right)}{2} \frac{\partial^{2} u }{\partial x^{2}}  + \frac{a \left(- 2 \Delta{t}^{2} a^{2} + 3 \Delta{t} \Delta{x} a - \Delta{x}^{2}\right)}{6} \frac{\partial^{3} u }{\partial x^{3}}  + \frac{a \left(- 6 \Delta{t}^{3} a^{3} + 12 \Delta{t}^{2} \Delta{x} a^{2} - 7 \Delta{t} \Delta{x}^{2} a + \Delta{x}^{3}\right)}{24} \frac{\partial^{4} u }{\partial x^{4}} $

```python
# display the amplification factor
DE1.display_amp_factor()
```

$\displaystyle e^{\Delta{t} \alpha} = - \frac{\Delta{t} a}{\Delta{x}} + \frac{\Delta{t} a e^{- i \Delta{x} k_{1}}}{\Delta{x}} + 1.0$

```python
# print the latex form of the generate_modified_equation
print(DE1.latex_modified_equation())
```

    \frac{\partial u }{\partial t}  =  - a \frac{\partial u }{\partial x}  + \frac{a \left(- \Delta{t} a + \Delta{x}\right)}{2} \frac{\partial^{2} u }{\partial x^{2}}  + \frac{a \left(- 2 \Delta{t}^{2} a^{2} + 3 \Delta{t} \Delta{x} a - \Delta{x}^{2}\right)}{6} \frac{\partial^{3} u }{\partial x^{3}}  + \frac{a \left(- 6 \Delta{t}^{3} a^{3} + 12 \Delta{t}^{2} \Delta{x} a^{2} - 7 \Delta{t} \Delta{x}^{2} a + \Delta{x}^{3}\right)}{24} \frac{\partial^{4} u }{\partial x^{4}}

```python
# print the latex form of the amplification factor
print(DE1.latex_amp_factor())
```

    e^{\Delta{t} \alpha} = - \frac{\Delta{t} a}{\Delta{x}} + \frac{\Delta{t} a e^{- i \Delta{x} k_{1}}}{\Delta{x}} + 1.0

Using the second method for constructing the rhs

```python
from pymodpde import DifferentialEquation, symbols, j, n # import the differential equation and the indicies we need to use

b = symbols('b') # define positive a constant for the advection velocity

DE2 = DifferentialEquation(dependentVarName='f',independentVarsNames=['y'], indices=[j])# construct the first upwind derivative of the dependent variabe 'f'
                                                                        # with respect to the independent variable 'y' using the stencil -1, and 0 evaluated at time tn = n

# method II of constructing the rhs:
advectionTerm2 = (DE2.f(time=n, y=j) - DE2.f(time=n, y = j-1)) / DE2.dy

DE2.set_rhs(- b * advectionTerm2 ) # set the rhs of the Differential equation DE

DE2.generate_amp_factor() # computing the amplification factor
```

```python
# display the amplification factor
DE2.display_amp_factor()
```

$\displaystyle e^{\Delta{t} \alpha} = - \frac{\Delta{t} b}{\Delta{y}} + \frac{\Delta{t} b e^{- i \Delta{y} k_{1}}}{\Delta{y}} + 1.0$

The difference equation for Backward in time and Upwind in space is
\begin{equation}
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=-a\left(\frac{u_{i}^{n+1}-u_{i-1}^{n+1}}{\Delta x}\right).
\end{equation}

```python
from pymodpde import DifferentialEquation, symbols

# define the advection velocity
a= symbols("a")
# define the spatial and temporal indices
i, n = symbols("i n")

# construct a time dependent differential equation
DE = DifferentialEquation(dependentVarName="u",independentVarsNames=["x"], indices=[i], timeIndex=n)

# method I of constructing the rhs:
advectionTerm = -a * DE.expr(order=1, directionName="x", time=n+1,stencil=[-1, 0])

# setting the rhs of the differential equation
DE.set_rhs( advectionTerm )

# computing and displaying the modified equation up to two terms
DE.generate_modified_equation(nterms=2)
```

```python
# display the mofified equation
DE.display_modified_equation()
```

$\displaystyle \frac{\partial u }{\partial t}  =  - a \frac{\partial u }{\partial x}  + \frac{a \left(\Delta{t} a + \Delta{x}\right)}{2} \frac{\partial^{2} u }{\partial x^{2}} $

```python
# display the amplification factor
DE.display_amp_factor()
```

$\displaystyle e^{\Delta{t} \alpha} = \frac{\Delta{x} e^{i \Delta{x} k_{1}}}{\Delta{t} a e^{i \Delta{x} k_{1}} - \Delta{t} a + \Delta{x} e^{i \Delta{x} k_{1}}}$

## Example Using BTCS on Advection-Diffusion PDEs (1D)
---

Consider the following PDE
\begin{equation}
u_t=-cu_x+\gamma u_{xx}
\end{equation}
to be solved using backward Euler discretization in time and central discretization in space(BTCS). The difference equation is
$$
\frac{u_{i}^{n+1}-u_{i}^{n}}{\Delta t}=-c\left(\frac{u_{i+1}^{n+1}-u_{i-1}^{n+1}}{2\Delta x}\right)+ \gamma \left(\frac{u_{i+1}^{n+1}-2u_i^{n+1}+u_{i-1}^{n+1}}{\Delta x^2}\right).
$$

```python
from pymodpde import DifferentialEquation, symbols, i, n # import the differential equation and the indicies we need to use

c, g = symbols('c gamma') # define positive a constant for the advection velocity

DE3 = DifferentialEquation(dependentVarName='u',independentVarsNames=['x'], indices=[i])# construct the first upwind derivative of the dependent variabe 'f'
                                                                        # with respect to the independent variable 'y' using the stencil -1, and 0 evaluated at time tn = n

# method II of constructing the rhs:
advectionTerm3 = (DE3.u(time=n+1, x=i+1) - DE3.u(time=n+1, x=i-1)) / DE3.dx /2

diffusionTerm3 = (DE3.u(time=n+1, x=i+1) - 2* DE3.u(time=n+1, x=i) + DE3.u(time=n+1, x=i-1)) / DE3.dx / DE3.dx

DE3.set_rhs(- c * advectionTerm3  ) # set the rhs of the Differential equation DE

DE3.generate_modified_equation(nterms=3)  # compute the modified equation up to 2 terms and display the latex form of the modified equation
```

```python
# display the mofified equation
DE3.display_modified_equation()
```

$\displaystyle \frac{\partial u }{\partial t}  =  - c \frac{\partial u }{\partial x}  + \frac{\Delta{t} c^{2}}{2} \frac{\partial^{2} u }{\partial x^{2}}  - \frac{c \left(2 \Delta{t}^{2} c^{2} + \Delta{x}^{2}\right)}{6} \frac{\partial^{3} u }{\partial x^{3}} $

```python
# display the amplification factor
DE3.display_amp_factor()
```

$\displaystyle e^{\Delta{t} \alpha} = \frac{2.0 \Delta{x} e^{i \Delta{x} k_{1}}}{\Delta{t} c e^{2.0 i \Delta{x} k_{1}} - \Delta{t} c + 2.0 \Delta{x} e^{i \Delta{x} k_{1}}}$

## Example Using FTUS on Advection PDEs (2D)
---

Consider the advection equation in a two dimensional space
\begin{equation}
u_{t}=-bu_{x}-cu_{y};\quad(b,c)\in\mathbb{R}^{+}
\end{equation}
to be discretized using a forward Euler method in time and UPWIND in space (FTUS)
\begin{equation}
\frac{u_{i,j}^{n+1}-u_{i,j}^{n}}{\Delta t}=-b\frac{u_{i,j}^{n}-u_{i-1,j}^{n}}{\Delta x}-c\frac{u_{i,j}^{n}-u_{i,j-1}^{n}}{\Delta y},
\end{equation}

with a modified equation

```python
from pymodpde import DifferentialEquation, symbols, i, j, n # import the differential equation and the indicies we need to use

b, c= symbols('b c') # define positive a constant for the advection velocity

DE1 = DifferentialEquation(dependentVarName='u',independentVarsNames=['x','y']) # construct a one D differential equation with 'x' and 'u' as independent and dependent variables, respectively

# method II of constructing the rhs:
advectionTerm1 = (DE1.u(time=n, x=i, y=j) - DE1.u(time=n, x=i-1, y=j))/DE1.dx

advectionTerm2 = (DE1.u(time=n, x=i, y=j) - DE1.u(time=n, x=i, y=j-1))/DE1.dy

# set the rhs of the differential equation
DE1.set_rhs( -b * advectionTerm1 - c * advectionTerm2 )

# compute and the modified equation up to two terms
DE1.generate_modified_equation(nterms=2)
```

```python
# display the mofified equation
DE1.display_modified_equation()
```

$\displaystyle \frac{\partial u }{\partial t}  =  - c \frac{\partial u }{\partial y}  - b \frac{\partial u }{\partial x}  + \frac{c \left(- \Delta{t} c + \Delta{y}\right)}{2} \frac{\partial^{2} u }{\partial y^{2}}  - \Delta{t} b c \frac{\partial^{2} u }{\partial y\partial x}  + \frac{b \left(- \Delta{t} b + \Delta{x}\right)}{2} \frac{\partial^{2} u }{\partial x^{2}} $

```python
# display the amplification factor
DE1.display_amp_factor()
```

$\displaystyle e^{\Delta{t} \alpha} = - \frac{\Delta{t} c}{\Delta{y}} + \frac{\Delta{t} c e^{- i \Delta{y} k_{2}}}{\Delta{y}} - \frac{\Delta{t} b}{\Delta{x}} + \frac{\Delta{t} b e^{- i \Delta{x} k_{1}}}{\Delta{x}} + 1.0$

### Numerical Solution

```python
nx = 64
ny = 64
Lx = 1.0
Ly = 1.0
x = np.linspace(0,Lx,nx)
y = np.linspace(0,Ly,ny)
dx = Lx/(nx-1)
dy = Ly/(ny-1)
# create a grid of coordinates
xx,yy = np.meshgrid(x,y)

u0 = lambda x,y : np.exp(-(x-0.2)**2/0.01)*np.exp(-(y-0.2)**2/0.01)

cx = 1.0
cy = 1.0

dt = 1/128
tend = 0.5 #s
t = 0

cflx = cx * dt/dx
cfly = cy * dt/dy

# setup the initial condition
sol = []
u = np.zeros([ny+2,nx+2]) # we will ghost cells to simplify the implementation of periodic BCs
u[1:-1, 1:-1] = u0(xx,yy)
# set periodic boundaries
u[:,0] = u[:,-3] #x-minus face
u[:,-1] = u[:,2] #x-plus face
u[0,:] = u[-3,:] #y-minus face
u[-1,:] = u[2,:] #y-plus face
sol.append(u)
```

```python
while t < tend:
    un = sol[-1]
    unew = un.copy()
    unew[1:-1,1:-1] = un[1:-1,1:-1] - cflx * (un[1:-1,1:-1] - un[1:-1,:-2]) - cfly * (un[1:-1,1:-1] - un[:-2,1:-1])
    # set periodic boundaries
    unew[:,0] = unew[:,-3] #x-minus face
    unew[:,-1] = unew[:,2] #x-plus face
    unew[0,:] = unew[-3,:] #y-minus face
    unew[-1,:] = unew[2,:] #y-plus face

    sol.append(unew)
    t += dt
```

```python
levs = np.linspace(0,1,15)
plt.figure(figsize=(10,8))
plt.contourf(xx,yy,sol[-1][1:-1,1:-1]+sol[0][1:-1,1:-1],cmap=cm.viridis,levels=levs,vmax=1.0,vmin=0)
plt.text(0.1, 0.4, 'time=0.0s', fontsize=12,family='serif')
plt.text(0.6, 0.92,'time={}s'.format(tend), fontsize=12,family='serif')
plt.xlabel('x', fontsize=14,family='serif')
plt.ylabel('y', fontsize=14,family='serif')
cbar = plt.colorbar()
plt.clim(-1,1)
cbar.set_ticks(np.linspace(-1,1,5))
plt.savefig('./advection_2d.pdf',transparent=True)
plt.show()
```

## Numerical Solver
---

```python
%matplotlib notebook
from numerical.layout import *
plty=PlotStyling()
```

    Tab(children=(VBox(children=(HBox(children=(FloatText(value=0.01, description='dt:'), FloatText(value=10.0, de…

```python

```


# Eigenvalues of $A$ based on $a$

```julia
using LinearAlgebra
function A(a)
    eigvals([1 a; a 1])
end

A(1)
```

    2-element Array{Float64,1}:
     0.0
     2.0

$
\begin{align}
det(A-\lambda I) &= 0 \\
(1-\lambda)^2 - a^2 &= 0 \\
\lambda^2 - 2 \lambda +  (1- a^2) &=0 \\
\rightarrow \lambda &= \frac{1 \pm \sqrt{1 - 1 +a^2}}{1} \\
&= 1 \pm \| a\|
\end{align}
$

# SVD and Eigenvalue

$k = \frac{\sigma_{max}}{\sigma_{min}}$

$a=1$ is making $A$ a singular matrix.

# Quadratic form

```julia
using Plots

function quadratic(u)
    range = collect(0:0.01:2)
    outpt = zeros(size(range,1))
    for n in 1:size(range,1)
        a = range[n]
        y =  u' * [1 a; a 1] * u
        outpt[n] = y
    end
    plot(range, outpt, xlabel="a", ylabel="f(u)"
        , title="u="*string(u), label=:false, linewidth=4)
end

quadratic([5, 2])
```

For $a=1$, the output of $f(u)$ function is $(u[1]+u[2])^2$;

## Adjourn

```julia
using Dates
println("mahdiar")
Dates.format(now(), "Y/U/d HH:MM")
```

    mahdiar

    "2021/February/11 17:54"