<form action="javascript:code_toggle()"><input type="submit" id="toggleButton" value="Show Code"></form>

```python
# Importing some python libraries.
import numpy as np
from numpy.random import randn
import matplotlib.pyplot as pl
import seaborn as sns
%matplotlib inline
# Fixing figure sizes
from pylab import rcParams
rcParams['figure.figsize'] = 10,5

import sympy as sp

pale_red = sns.xkcd_rgb['pale red']
denim_blue = sns.xkcd_rgb['denim blue']
```

    /home/kostis/anaconda/lib/python2.7/site-packages/matplotlib/__init__.py:872: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
      warnings.warn(self.msg_depr % (key, alt_key))

# Overdamped Langevin Equation

The overdamped Langevin equation is defined as

$$
dX_t=-\nabla U(x)dt+\sigma dB_t,
$$
for some potential $U$. $B_t$ represents Brownian motion and $\sigma$ controls the "strength" of the random variations.

In this example, we will work with a specific potential

$$
U(x)=(b-a/2)(x^2-1)^2+a/2\cdot (x+1).
$$

This is a double-well potential, as can be seen in the following plot.

```python
a = -1;
b = 1;

def U(x,a=-1,b=1):
    return (b-a/2)*(x**2-1)**2+a/2*(x+1)

x = np.linspace(-1.5,1.5)
pl.plot(x,U(x),color=pale_red,linewidth=5)
pl.title('The potential $U(x)$ with $a=-1$ and $b=1$',fontsize=20)
```

## Deterministic System

Let's write the equation down, given the specific potential. First, if $\sigma=0$,then the equation is an ODE

$$
\begin{align}
\frac{dX_t}{dt}&=4c(1-x^2)x-a/2,\\c&=(b-a/2).
\end{align}
$$

```python
# Defining the derivative of the potential
def Uprime(x,t,a=-1,b=1):
    return 4*(b-a/2.0)*x*(1-x**2)-a/2.0
```

By numerically solving $U'(x)=0$, we can find that there are three equilibrium points for the system. Approximately, those are

$$
\begin{align}
x_1&=-0.955393,\\
x_2&=-0.083924,\\
x_3&=1.03932.
\end{align}
$$

```python
from scipy.integrate import odeint # importing a solver

t   = np.linspace(0,10,100)

xinit = np.array([2.0,1.0,-0.08,-0.9,-2])

with sns.cubehelix_palette(3):
    for i in xrange(5):
        sol = odeint(Uprime, xinit[i], t)
        pl.plot(t,sol,alpha=0.8,linewidth=10)

pl.title('Five different solutions of the ODE system',fontsize=20)
pl.xlabel('t',fontsize=20)
```

As we can see, out of the three equilibrium solutions of the system, the two are stable and the one in the middle is unstable. We will use this information for comparisons with the stochastic system.

# Stochastic System

Let us now assume that $\sigma>0$. In that case, we have an SDE, which we can solve with Euler-Maruyama. The scheme shall be :

$$
X_{n+1}=X_{n}+f(X_n)\Delta t+\sigma \sqrt{\Delta t}\cdot z,
$$

where $z$ is a standard normal distribution.

```python
def EM(xinit,sigma,T,Dt=0.1,a=-1,b=1):
    '''
        Returns the solution of the Langevin equation with
        potential U.

        Arguments
        =========
        xinit : real, initial condition.
        sigma : real, standard deviation parameter, used in generating brownian motion.
        Dt    : real, stepsize of the Euler-Maruyama.
        T     : real, final time to reach.

    '''

    n = int(T/Dt) # number of steps to reach T
    X = np.zeros(n)
    z = sigma*randn(n)

    X[0] = xinit # Initial condition

    # EM method
    for i in xrange(1,n):
        X[i] = X[i-1] + Dt* Uprime(X[i-1],a,b) + np.sqrt(Dt)*z[i-1]

    return X

```

Now we can reproduce the picture from the deterministic case, but this time with the extra stochastic part. When $\sigma$ is small, we see a similar picture with previously as the deterministic dynamics overpower the stochasticity.

```python
with sns.cubehelix_palette(3):
    for i in xrange(5):
        path = EM(xinit[i],sigma=0.1,T=10)
        pl.plot(t,path,alpha=0.7,linewidth=10)

pl.title('Trajectories of the Langevin SDE, $\sigma=0.1$',fontsize=20)
pl.xlabel('t',fontsize=20)
```

```python
with sns.cubehelix_palette(5):
    for i in xrange(5):
        path = EM(xinit[i],sigma=0.4,T=10)
        pl.plot(t,path,alpha=0.9,linewidth=5)

pl.title('Trajectories of the Langevin SDE, $\sigma=0.4$',fontsize=20)
pl.xlabel('t',fontsize=20)
```

Changing the $\sigma$ from $0.1$ to $0.4$ provides random "kicks" that are hard enough for the solutions to jump from one equilibrium to the other.

```python
with sns.cubehelix_palette(5):
    for i in xrange(5):
        path = EM(xinit[i],sigma=1,T=10)
        pl.plot(t,path,alpha=0.7,linewidth=5)

pl.title('Trajectories of the Langevin SDE, $\sigma=1$',fontsize=20)
pl.xlabel('t',fontsize=20)

```

With $\sigma=1$, the trajectories can now move freely between the equilibrium points. We can still see some kind of attraction though to the area around them.

Let us attempt to set $\sigma$ to a larger number and see what happens.

```python
with sns.cubehelix_palette(5):
    for i in xrange(5):
        path = EM(xinit[i],sigma=2.4,T=10)
        pl.plot(t,path,linewidth=5)

pl.title('Trajectories of the Langevin SDE, $\sigma=1$',fontsize=20)
pl.xlabel('t',fontsize=20)
```

Now the kicks are strong enough that the attractiveness (or repulsiveness) of the stationary points looks completely irrelevant. The dynamics are all about the stochastic part.

## Changing the properties of the potential

We now fix $\sigma=1$ and look at the paths for $a\in [0,b]$. We will start from $X_0=1.3$.

```python
b = 1
arange = np.linspace(0,b,3)

with sns.cubehelix_palette(3):
    for aval in arange:
        path = EM(1.3,sigma=1, T=10, a=aval)
        pl.plot(t,path,linewidth=4)

pl.title('With $a=0,0.5,1$',fontsize=20)
pl.xlabel('$t$',fontsize=20)
```

```python
b = 1
arange = np.linspace(0,b,3)

with sns.cubehelix_palette(3):
    for aval in arange:
        pl.plot(x,U(x,a=aval),linewidth=4)

pl.title('With $a=0,0.5,1$',fontsize=20)
pl.xlabel('$x$',fontsize=20)
```

Above we have the potentials corresponding to the paths. To make things more concrete, here is the path superimposed on the potential.

```python
def plotOnPoten(a,T=2):
    x = np.linspace(-2,2)
    pl.plot(x,U(x,a=0.5),linewidth=1,color='black')
    path = EM(1.3,sigma=1, T=T, a=aval)

    pl.plot(path,U(path,a=0.5),linewidth=4,alpha=0.7,color=pale_red)
    pl.title('For $a='+str(a)+'$.',fontsize=20)
```

```python
plotOnPoten(1,T=10)
```


# <center> ЛАБОРАТОРНАЯ  РАБОТА 5 </center>
# <center> РЕШЕНИЕ  СИСТЕМ  ЛИНЕЙНЫХ АЛГЕБРАИЧЕСКИХ УРАВНЕНИЙ ИТЕРАЦИОННЫМИ МЕТОДАМИ   </center>

Теоретический материал к данной теме содержится в [1, глава 6], [2, часть 2, глава 2].

```python
from sympy.solvers import solve
from sympy import exp, sin, cos, sqrt, log, ln, pi
from sympy import Rational as syR
from sympy import Symbol, diff
from scipy.misc import derivative

import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline
```

## Задача 1.
Дана  система уравнений $Ax=b$. Найти решение  системы  с помощью метода Гаусса. Выполнить 10 итераций по методу Зейделя. Принимая решение,  полученное с помощью метода Гаусса  за точное, найти величину абсолютной погрешности итерационного решения.
## Порядок решения задачи:
1. Задать матрицу системы $A$ и вектор правой части $b$. Используя встроенную функцию, найти решение системы $Ax=b$ с помощью метода Гаусса.
2. Преобразовать систему $Ax=b$ к виду $x=Bx+c$, удобному для итераций. Проверить выполнение достаточного условия сходимости итерационных методов (модуль по норме не превышает 1).
3. Реализовать функцию zeid, описывающую метод Зейделя. Выполнить 10  итераций по методу Зейделя; взять любое начальное приближение. Принимая решение, полученное в п. 1 за точное, найти величину абсолютной погрешности итерационного решения (использовать норму $|| .||_{\infty}$).
4. Взять другое начальное приближение. Объяснить полученные результаты.

Ваши матрица A и вектор b:
$A = {Task1[TASK_VARIANT][A_latex]}$
$b = {Task1[TASK_VARIANT][b_latex]}$

```python
epsilon1 = 10**(-6)
# Матрица и вектор
A = {Task1[TASK_VARIANT][A_latex]}
b = {Task1[TASK_VARIANT][b_latex]}
```

Найдите решение системы $Ax=b$ с помощью встроенной функции (например,np.linalg.solve).

```python
##массив-ответ на систему
```

Реализуйте функцию для нахождения решения с помощью метода Гаусса.

```python
def meth_gauss
```

С помощью написанной ранее функции найдите найдите решения Вашей системы.

```python
##массив-ответ
```

Преобразуйте систему $Ax=b$ к виду $x=Bx+c$. Для этого определите функции для построения B и c.

```python
##место для кода
```

Проверьте выполнение достаточного условия сходимости итерационного метода. Можно осуществить с помощью встроенной функции(например, np.linalg.norm).

```python
##ответ число меньше 1
```

Реализуйте метод Зейделя.

```python
def zeid(B, c, x0, N):
```

## Задача 2.
Для системы уравнений $Ax=b$ из задачи 1 найти  решение  по методу Зейделя с  точностью $\varepsilon = 10^{-6}$, взяв любое начальное приближение. Для этого  модифицировать функцию zeid  так, чтобы  решение вычислялось с заданной точностью $\varepsilon$. Предусмотреть подсчет  количества итераций, потребовавшихся для достижения точности $\varepsilon$.

Реализуйте функцию zeideps с возможностью вычислять решение с определенной точностью.

```python
def zeideps(A, b, x0, eps):
```

```python

```

#### ЛИТЕРАТУРА:
1. Амосов А.А., Дубинский Ю.А., Копченова Н.В. Вычислительные методы для инженеров. М.: Высшая школа, 1994.
2. Самарский А.А., Гулин А.В. Численные методы. М.: Наука, 1989.

```python

```


For this problem set, we'll be using the Jupyter notebook:

## Four numbers

Four numbers are consecutive multiples of 8 and their sum is 336.

What are the numbers?

Use brute force if possible!

Other solution can be:

    import sympy
    x = sympy.var('x')
    s = sympy.solve(8*x+8*(x+1)+8*(x+2)+8*(x+3)-336,x)[0]
    numbers = [8*(s+i) for i in range(4)]

```python
numbers = []
### BEGIN SOLUTION
for s in range(20):
    numbers = [(i+s)*8 for i in range(4)]
    print(numbers,':::' ,sum(numbers))

    if sum(numbers) == 336:
        break
### END SOLUTION
```

    [0, 8, 16, 24] ::: 48
    [8, 16, 24, 32] ::: 80
    [16, 24, 32, 40] ::: 112
    [24, 32, 40, 48] ::: 144
    [32, 40, 48, 56] ::: 176
    [40, 48, 56, 64] ::: 208
    [48, 56, 64, 72] ::: 240
    [56, 64, 72, 80] ::: 272
    [64, 72, 80, 88] ::: 304
    [72, 80, 88, 96] ::: 336

```python
assert sum(numbers) == 336
assert len(numbers) == 4

for i in range(1,4):
    assert numbers[i]-numbers[i-1] == 8

```


# Time Series Modelling

We follow the notations of Tsay R. (2010) *Analysis of Financial Time Series*. John Wiley & Sons, Inc., Hoboken, New Jersey.

The following notebook is an adaptation of Tsay's book except the codes are written in Python

## Normal Distribution

A traditional assumption made in financial study is that the simple returns ${R_{it} | t = 1, . . . , T }$ are independently and identically distributed as normal with fixed mean and variance. This assumption makes statistical properties of asset returns tractable. But it encounters several difficulties.

* The lower bound of a simple return is −1. Yet the normal distribution may assume any value in the real line and, hence, has no lower bound.
* If $R_{it}$ is normally distributed, then the multiperiod simple return $R_{it}[k]$ is not normally distributed because it is a product of one-period returns.
* The normality assumption is not supported by many empirical asset returns, which tend to have a positive excess kurtosis.

## Lognormal Distribution

Another commonly used assumption is that the log returns $r_t$ of an asset are independent and identically distributed (iid) as normal with mean $\mu$ and variance $\sigma^2$. The simple returns are then iid lognormal random variables with mean and variance given by

\begin{align}
E(R_t) & = \text{exp}(\mu+\frac{\sigma^2}{2})-1 \\
\text{Var}(R_t) & =\text{exp}(2\mu+\sigma^2)[\text{exp}(\sigma^2)-1] \\
\end{align}

These two equations are useful in studying asset returns (e.g., in forecasting using models built for log returns).

Alternatively, let $m_1$ and $m_2$ be the mean and variance of the simple return $R_t$ , which is lognormally distributed. Then the mean and variance of the corresponding log return $r_t$ are

\begin{align}
E(r_t) & = ln\bigg[ \frac{m_1+1}{\sqrt{1+ \frac{m_2}{(1+m_1)^2}}}\bigg] \\
\text{Var}(r_t) & =ln\bigg[ 1 + \frac{m_2}{(1 + m_1)^2}\bigg] \\
\end{align}

Because the sum of a finite number of iid normal random variables is normal, $r_t[k]$ is also normally distributed under the normal assumption for ${r_t}$. In addition, there is no lower bound for $r_t$ , and the lower bound for $R_t$ is satisfied using
$1 + R_t = \text{exp}(r_t)$. However, the lognormal assumption is not consistent with all the properties of historical stock returns. In particular, many stock returns exhibit a positive excess kurtosis.

***

## Stable Distribution

The stable distributions are a natural generalization of normal in that they are stable under addition, which meets the need of continuously compounded returns $r_t$. Furthermore, stable distributions are capable of capturing excess kurtosis shown
by historical stock returns. However, nonnormal stable distributions do not have a finite variance, which is in conflict with most finance theories. In addition, statistical modeling using nonnormal stable distributions is difficult. An example of
nonnormal stable distributions is the Cauchy distribution, which is symmetric with respect to its median but has infinite variance.

## Scale Mixture of Normal Distributions

Recent studies of stock returns tend to use scale mixture or finite mixture of normal distributions. Under the assumption of scale mixture of normal distributions, the log return $r_t$ is normally distributed with mean $\mu$ and variance $\sigma^2$ [i.e., $r_t ∼ N(\mu, \sigma^2)$]. However, $\sigma^2$ is a random variable that follows a positive distribution (e.g., $\sigma^{-2}$ follows a gamma distribution). An example of finite mixture of normal distributions is

$$r_t ∼ (1-X)N(\mu,\sigma_1^2)+XN(\mu,\sigma_2^2)$$

where $X$ is a Bernoulli random variable such that $P(X = 1) = \alpha$ and $P(X = 0) = 1 − \alpha$ with $0 < \alpha < 1$, $\sigma_1^2$ is small, and $\sigma_2^2$ is relatively large. For instance, with $\alpha = 0.05$, the finite mixture says that 95% of the returns follow $N(\mu, \sigma_1^2)$ and 5% follow $N(\mu, \sigma_2^2)$. The large value of $\sigma_2^2$ enables the mixture to put more mass at the tails of its distribution. The low percentage of returns that are from $N(\mu, \sigma_2^2)$ says
that the majority of the returns follow a simple normal distribution. Advantages of mixtures of normal include that they maintain the tractability of normal, have finite higher order moments, and can capture the excess kurtosis. Yet it is hard to
estimate the mixture parameters (e.g., the $\alpha$ in the finite-mixture case).

[Jake VanderPlas - Kernel Density Estimation in Python](https://jakevdp.github.io/blog/2013/12/01/kernel-density-estimation/)

[Jake VanderPlas - Kernel Density Estimation in Python Notebook](http://nbviewer.jupyter.org/url/jakevdp.github.com/downloads/notebooks/KDEBench.ipynb)

[Michael Lerner - Histograms and kernel density estimation KDE 2](https://mglerner.github.io/posts/histograms-and-kernel-density-estimation-kde-2.html?p=28)

```python
import math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
sns.set()
from pandas_datareader import data, wb
import datetime
from scipy import stats
```

```python
fig, ax = plt.subplots(1, 1, figsize=(12,6))
x = np.linspace(-4, 4, 100)
line1, = ax.plot(x, stats.norm.pdf(x), 'k-', lw=2, alpha=0.8, label='normal pdf')
line2, = ax.plot(x, stats.cauchy.pdf(x), 'r-', lw=2, alpha=0.8, label='cauchy pdf')
plt.legend([line1, line2], ["Normal", "Cauchy"]);
```
from sklearn.neighbors import KernelDensity

fig, ax = plt.subplots(figsize=(8, 6))
ax.grid(color='white', linestyle='-', linewidth=2)
X_src = np.zeros((1, 1))
x_grid = np.linspace(-4, 4, 1000)
log_dens = KernelDensity(kernel='gaussian').fit(X_src).score_samples(x_grid[:, None])
ax.plot(x_grid, np.exp(log_dens), lw=3, alpha=0.5, label='gaussian');from sklearn.mixture import GaussianMixturegmm = GaussianMixture(n_components=2, max_iter=1000,
                      weights_init = [0.9, 0.1],
                      init_params='random')

samples_1 = np.random.normal(0, 1, 10000)
samples_2 = np.random.normal(0, 2, 10000)
X = np.concatenate((samples_1, samples_2))

gmm = gmm.fit(X.reshape(-1,1))
gmm_x = np.linspace(-4, 4, 1000)
gmm_y = np.exp(gmm.score_samples(gmm_x.reshape(-1, 1)))
plt.plot(gmm_x, gmm_y);def gen_mix(x, mu, sigma, weights):
    gmm = GaussianMixture(n_components=2, max_iter=1000,
                          weights_init = weights,
                          covariance_type = 'full',
                          init_params = 'random')

    samples_1 = np.random.normal(mu[0], sigma[0], 100000)
    samples_2 = np.random.normal(mu[1], sigma[1], 100000)
    print(mu[1], sigma[1])
    X = np.concatenate((samples_1, samples_2))

    gmm = gmm.fit(X.reshape(-1, 1))
    gmm_x = np.linspace(-4, 4, 1000)
    gmm_y = np.exp(gmm.score_samples(gmm_x.reshape(-1, 1)))

    return gmm_x, gmm_y

```python
def std_norm_pdf(x):
    return 1 / np.sqrt(2 * math.pi) * np.exp(-x ** 2 / 2)
```

```python
def norm_pdf(mu, sigma, x):
    return 1 / (sigma * np.sqrt(2 * math.pi)) * np.exp(- 1 / 2 * ((x - mu) / sigma) ** 2)
```

```python
mu = [0, 0]
sigma = [1, 4]
alpha = 0.05
length = 100000

x = np.linspace(-4, 4, length)

samples_1 = np.random.normal(mu[0], sigma[0], length)
samples_2 = np.random.normal(mu[1], sigma[1], length)
ind = np.random.rand(length)
res = []
for k, v in enumerate(ind):
    if v >= alpha:
        res.append(samples_1[k])
    else:
        res.append(samples_2[k])
mm_mean = np.mean(res)
mm_std = np.std(res)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(12,6))
x = np.linspace(-4, 4, length)
y = norm_pdf(mm_mean, mm_std, x)

line1, = ax.plot(x, stats.norm.pdf(x), 'k-', lw=2, alpha=0.8, label='Nrmal')
line2, = ax.plot(x, stats.cauchy.pdf(x), 'r-', lw=2, alpha=0.8, label='Cauchy')
line3, = ax.plot(x, y, 'g-', lw=1, alpha=0.8, label='Mixture')
plt.legend([line1, line2, line3], ["Normal", "Cauchy", "Mixture"]);
```

# Multivariate Returns

Joint distribution

How conditional expectation and conditional covariance matrix evolve over time

Random Vector $X = (X_1, ..., X_p)$

$E(X)=\mu_x=[E(X_1), ..., E(X_p)]^\prime$

$Cov(X)=\sum_x=E[(X - \mu_x)(X - \mu_x)^\prime]$

# Likelihood Function of Returns

Some financial theories such as the capital asset pricing model (CAPM) of Sharpe (1964) focus on the joint distribution of $N$ returns at a **single time index $t$** (i.e., the distribution of ${r_{1t}, ..., r_{Nt}}$). In this, we are looking at the returns of all the different assets, $1, ..., N$, at time $T$ only.

Other theories emphasize the **dynamic structure** of individual asset returns (i.e., the distribution of {$r_{i1}, ..., r_{iT}$} for a given asset $i$). Here are only looking at the returns of asset $i$ across time, $1, ..., T$.

In univariate analysis, our main concern is the joint distribution of {$r_{it}$}$^T_{t=1}$ for asset $i$. To this end, it is useful to partition the joint distribution as

\begin{align}
F(r_{i1}, ...r_{iT}; \theta) & = F(r_{i1}; \theta)F(r_{i2}|r_{i1}; \theta)...F(r_{iT}|r_{i, T-1}, ...,r_{i1}; \theta) \\
& = F(r_{i1}; \theta)\prod_{t=2}^TF(r_{it}|r_{i,t-1}, ..., r_{i1}; \theta) \\
\end{align}

This partition highlights the temporal dependencies of the log return $r_{it}$.

The above equation can be used to obtain the likelihood function of the log returns {$r_1, ..., r_T$} of an asset.

If the conditional distribution $f(r_t|r_{t-1}, ..., r_1, \theta)$ is normal with mean $\mu_t$ and variance $\sigma_t^2$, when $\theta$ consists of the parameters in $\mu_t$ and $\sigma_t^2$ and the likelihood function of the data is:

$$f(r_1, ..., r_T;\theta)=f(r_1;\theta)\prod_{t=2}^T\frac{1}{\sqrt{2\pi}\sigma_t}exp\bigg[\frac{-(r_t-\mu_t)^2}{2\sigma_t^2}\bigg]$$

where $f(r_1; θ)$ is the marginal density function of the first observation $r_1$. The value of $\theta$ that maximizes this likelihood function is the maximum-likelihood estimate (MLE) of $\theta$. Since the log function is monotone, the MLE can be obtained by maximizing the log-likelihood function,

$$ln\  f(r_1, ..., r_T;\theta)=ln\ f(r_1;\theta) - \frac{1}{2} \sum_{t=2}^T\bigg[ln(2\pi)+ln(\sigma_t^2)+\frac{(r_t-\mu_t)^2}{\sigma_t^2}\bigg]$$

# Empirical Properties of Return

```python
start = datetime.datetime(1975, 1, 1)
end = datetime.datetime(2015, 1, 1)
```

```python
f = data.DataReader("IBM", 'yahoo', start, end)
```

```python
df = f.resample('M').asfreq().ffill()
```

```python
df.head()
```

      Open
      High
      Low
      Close
      Adj Close
      Volume

      Date

      1974-12-31
      10.375000
      10.554688
      10.375000
      10.500000
      3.329909
      782400.0

      1975-01-31
      11.453125
      11.812500
      11.453125
      11.765625
      3.731280
      1542400.0

      1975-02-28
      13.265625
      13.484375
      13.242188
      13.468750
      4.303698
      1094400.0

      1975-03-31
      12.968750
      13.031250
      12.859375
      12.914063
      4.126455
      796800.0

      1975-04-30
      12.640625
      13.125000
      12.609375
      13.117188
      4.191364
      1099200.0

```python
df['Simple Return'] = df['Adj Close'].pct_change()
df.head()
```

      Open
      High
      Low
      Close
      Adj Close
      Volume
      Simple Return

      Date

      1974-12-31
      10.375000
      10.554688
      10.375000
      10.500000
      3.329909
      782400.0
      NaN

      1975-01-31
      11.453125
      11.812500
      11.453125
      11.765625
      3.731280
      1542400.0
      0.120535

      1975-02-28
      13.265625
      13.484375
      13.242188
      13.468750
      4.303698
      1094400.0
      0.153411

      1975-03-31
      12.968750
      13.031250
      12.859375
      12.914063
      4.126455
      796800.0
      -0.041184

      1975-04-30
      12.640625
      13.125000
      12.609375
      13.117188
      4.191364
      1099200.0
      0.015730

```python
df['Log Return'] = np.log(df['Simple Return'] + 1)
df.head()
```

      Open
      High
      Low
      Close
      Adj Close
      Volume
      Simple Return
      Log Return

      Date

      1974-12-31
      10.375000
      10.554688
      10.375000
      10.500000
      3.329909
      782400.0
      NaN
      NaN

      1975-01-31
      11.453125
      11.812500
      11.453125
      11.765625
      3.731280
      1542400.0
      0.120535
      0.113806

      1975-02-28
      13.265625
      13.484375
      13.242188
      13.468750
      4.303698
      1094400.0
      0.153411
      0.142723

      1975-03-31
      12.968750
      13.031250
      12.859375
      12.914063
      4.126455
      796800.0
      -0.041184
      -0.042056

      1975-04-30
      12.640625
      13.125000
      12.609375
      13.117188
      4.191364
      1099200.0
      0.015730
      0.015608

```python
#fig = plt.subplots(2,1, figsize=(10,8), sharey=True)
fig = plt.figure(figsize=(10,8))
ax1 = fig.add_subplot(211)
ax2 = fig.add_subplot(212)
fig.suptitle('Time plots of monthly returns of IBM stock from January 1976 to Jan 2015 \n \
Upper panel is for simple returns, and lower panel is for log returns')

y_min = df[['Simple Return', 'Log Return']].min().min()
y_max = df[['Simple Return', 'Log Return']].max().max()
ax1.set_ylim([y_min, y_max])
ax1.plot(df['Simple Return']);
ax1.axes.set_ylabel('Simple Return');

ax2.set_ylim([y_min, y_max])
ax2.plot(df['Log Return'])
ax2.axes.set_ylabel('Log Return');
```

***

Time Series Modelling with Python 2.ipynb


# Load packages

```python
import numpy as np
from sympy import *
from sympsi import *

from sympy.physics.secondquant import F
from sympy.physics.quantum.dagger import Dagger
from sympy.physics.quantum import TensorProduct
from sympy.physics.quantum import Operator
from sympy.matrices import Matrix, banded

# This is required to get around a bug in sympy
Matrix.adjoint = lambda self: self.T.applyfunc(Dagger)

```

```python
# We start with creating the full pairing Hamiltonian for N sites
N = 4

a = symbols(f"a_:{N}")
a = [Operator(ai) for ai in a]
```

```python
# We construct the vector containing the annihilation and creation operators for fermions on each site
# This follows the explanation at https://topocondmat.org/w1_topointro/1D.html
c = Matrix(a + [Dagger(ai) for ai in a])
c
```

$\displaystyle \left[\begin{matrix}a_{0}\\a_{1}\\a_{2}\\a_{3}\\a_{0}^{\dagger}\\a_{1}^{\dagger}\\a_{2}^{\dagger}\\a_{3}^{\dagger}\end{matrix}\right]$

```python
# Check whether taking the hermitin conjugate works properly
Dagger(c)
```

$\displaystyle \left[\begin{matrix}a_{0}^{\dagger} & a_{1}^{\dagger} & a_{2}^{\dagger} & a_{3}^{\dagger} & a_{0} & a_{1} & a_{2} & a_{3}\end{matrix}\right]$

```python
# We create the Bogoliubov Hamiltonian matrix
epsilon = symbols(f"epsilon_:{N}")
delta = symbols('Delta')
E = banded({0: epsilon})
C = banded({-1: (N-1)*[-delta], 1: (N-1)*[delta]})

G = Matrix([[E,C], [-C,-E]])
G
```

$\displaystyle \left[\begin{matrix}\epsilon_{0} & 0 & 0 & 0 & 0 & \Delta & 0 & 0\\0 & \epsilon_{1} & 0 & 0 & - \Delta & 0 & \Delta & 0\\0 & 0 & \epsilon_{2} & 0 & 0 & - \Delta & 0 & \Delta\\0 & 0 & 0 & \epsilon_{3} & 0 & 0 & - \Delta & 0\\0 & - \Delta & 0 & 0 & - \epsilon_{0} & 0 & 0 & 0\\\Delta & 0 & - \Delta & 0 & 0 & - \epsilon_{1} & 0 & 0\\0 & \Delta & 0 & - \Delta & 0 & 0 & - \epsilon_{2} & 0\\0 & 0 & \Delta & 0 & 0 & 0 & 0 & - \epsilon_{3}\end{matrix}\right]$

```python
# Next we show that the Bogoliubov matrix is indeed like the original matrix
H = Dagger(c)*G*c
h1 = H[0]
h1
```

$\displaystyle \left(- \Delta a_{1}^{\dagger} - \epsilon_{0} a_{0}\right) a_{0}^{\dagger} + \left(\Delta a_{2}^{\dagger} - \epsilon_{3} a_{3}\right) a_{3}^{\dagger} + \left(\Delta a_{1} + \epsilon_{0} a_{0}^{\dagger}\right) a_{0} + \left(- \Delta a_{2} + \epsilon_{3} a_{3}^{\dagger}\right) a_{3} + \left(\Delta a_{0}^{\dagger} - \Delta a_{2}^{\dagger} - \epsilon_{1} a_{1}\right) a_{1}^{\dagger} + \left(\Delta a_{1}^{\dagger} - \Delta a_{3}^{\dagger} - \epsilon_{2} a_{2}\right) a_{2}^{\dagger} + \left(- \Delta a_{0} + \Delta a_{2} + \epsilon_{1} a_{1}^{\dagger}\right) a_{1} + \left(- \Delta a_{1} + \Delta a_{3} + \epsilon_{2} a_{2}^{\dagger}\right) a_{2}$

```python
# We will now continue showing that the this big matrix can actually be written as the sum of a smaller matrix of only two sites
# To do this we actually first create the system for two sites with an undefined Bogoliubov matrix
Nr = 2

ar = symbols(f"a_:{Nr}")
ar = [Operator(ai) for ai in ar]

cr = Matrix(ar + [Dagger(ai) for ai in ar])

xr = np.array(symbols(f"x_:{2*Nr}:{2*Nr}")).reshape((2*Nr,2*Nr))
Gr = Matrix(xr)

# Execute the formulism to get the Hamiltonian in reduced space
Hr = Dagger(cr)*Gr*cr
h2 = Hr[0]
h2
```

$\displaystyle \left(x_{00} a_{0}^{\dagger} + x_{10} a_{1}^{\dagger} + x_{20} a_{0} + x_{30} a_{1}\right) a_{0} + \left(x_{01} a_{0}^{\dagger} + x_{11} a_{1}^{\dagger} + x_{21} a_{0} + x_{31} a_{1}\right) a_{1} + \left(x_{02} a_{0}^{\dagger} + x_{12} a_{1}^{\dagger} + x_{22} a_{0} + x_{32} a_{1}\right) a_{0}^{\dagger} + \left(x_{03} a_{0}^{\dagger} + x_{13} a_{1}^{\dagger} + x_{23} a_{0} + x_{33} a_{1}\right) a_{1}^{\dagger}$

```python
# We will now match the coefficients
# First step: create a library of coefficients of the full hamiltonian
lib = {e.args[-1]: e/e.args[-1] for e in h1.args}
lib
```

    {Dagger(a_3): Delta*Dagger(a_2) - epsilon_3*a_3,
     a_0: Delta*a_1 + epsilon_0*Dagger(a_0),
     a_3: -Delta*a_2 + epsilon_3*Dagger(a_3),
     Dagger(a_0): -Delta*Dagger(a_1) - epsilon_0*a_0,
     Dagger(a_1): Delta*Dagger(a_0) - Delta*Dagger(a_2) - epsilon_1*a_1,
     Dagger(a_2): Delta*Dagger(a_1) - Delta*Dagger(a_3) - epsilon_2*a_2,
     a_1: -Delta*a_0 + Delta*a_2 + epsilon_1*Dagger(a_1),
     a_2: -Delta*a_1 + Delta*a_3 + epsilon_2*Dagger(a_2)}

```python
# Now, for each term in the reduced Hamiltonian, we will match the element of the terms in the reduced and full Hamiltonian
Gr1 = Gr.copy()
for i in range(2*Nr):
    # get the operator of the term
    term, ci = h2.args[i].args

    # create a lookup for operators and elements for this particular term
    store = {e.args[-1]: e/e.args[-1] for e in ncollect(lib[ci], cr).args}

    # for each operator in the term, match with the full Hamiltonian and substitute
    for term in ncollect(term, cr).args:
        xii, ki = term.args
        ki = store.get(ki, 0)
        Gr1 = Gr1.subs(xii, ki)
Gr1
```

$\displaystyle \left[\begin{matrix}\epsilon_{0} & 0 & 0 & \Delta\\0 & \epsilon_{1} & - \Delta & 0\\0 & - \Delta & - \epsilon_{0} & 0\\\Delta & 0 & 0 & - \epsilon_{1}\end{matrix}\right]$

OK, great! We have a reduced matrix for the first two sites. Next we need to check whether the matrix of the next two sites is structually identical.

```python
# We will now continue showing that the this big matrix can actually be written as the sum of a smaller matrix of only two sites
# To do this we actually first create the system for two sites with an undefined Bogoliubov matrix
ar = symbols(f"a_:{N}")
ar = ar[2:]
ar = [Operator(ai) for ai in ar]

cr = Matrix(ar + [Dagger(ai) for ai in ar])

xr = np.array(symbols(f"x_:{2*Nr}:{2*Nr}")).reshape((2*Nr,2*Nr))
Gr = Matrix(xr)

# Execute the formulism to get the Hamiltonian in reduced space
Hr = Dagger(cr)*Gr*cr
h2 = Hr[0]

# Now, for each term in the reduced Hamiltonian, we will match the element of the terms in the reduced and full Hamiltonian
for i in range(2*Nr):
    # get the operator of the term
    term, ci = h2.args[i].args

    # create a lookup for operators and elements for this particular term
    store = {e.args[-1]: e/e.args[-1] for e in ncollect(lib[ci], cr).args}

    # for each operator in the term, match with the full Hamiltonian and substitute
    for term in ncollect(term, cr).args:
        xii, ki = term.args
        ki = store.get(ki, 0)
        Gr = Gr.subs(xii, ki)
Gr
```

$\displaystyle \left[\begin{matrix}\epsilon_{2} & 0 & 0 & \Delta\\0 & \epsilon_{3} & - \Delta & 0\\0 & - \Delta & - \epsilon_{2} & 0\\\Delta & 0 & 0 & - \epsilon_{3}\end{matrix}\right]$

It is!
We can now proceed and check whether the sum of these two is actually the same as the full Hamiltonian

```python
# We will now continue showing that the this big matrix can actually be written as the sum of a smaller matrix of only two sites
# To do this we actually first create the system for two sites with an undefined Bogoliubov matrix
result = Add()
for i in range(0, N-1, 1):
    ar = a[i:i+2]
    #print(ar)
    cr = Matrix(ar + [Dagger(ai) for ai in ar])

    xr = np.array(symbols(f"x_:{4}:{4}")).reshape((4,4))
    Gr = Matrix(xr)

    # Execute the formulism to get the Hamiltonian in reduced space
    Hr = Dagger(cr)*Gr*cr
    h2 = Hr[0]

    # Now, for each term in the reduced Hamiltonian, we will match the element of the terms in the reduced and full Hamiltonian
    for i in range(2*Nr):
        # get the operator of the term
        term, ci = h2.args[i].args

        # create a lookup for operators and elements for this particular term
        store = {e.args[-1]: e/e.args[-1] for e in ncollect(lib[ci], cr).args}

        # for each operator in the term, match with the full Hamiltonian and substitute
        for term in ncollect(term, cr).args:
            xii, ki = term.args
            ki = store.get(ki, 0)
            Gr = Gr.subs(xii, ki)

    result += (Dagger(cr)*Gr*cr)[0]
result
```

$\displaystyle \left(\Delta a_{0}^{\dagger} - \epsilon_{1} a_{1}\right) a_{1}^{\dagger} + \left(- \Delta a_{1}^{\dagger} - \epsilon_{0} a_{0}\right) a_{0}^{\dagger} + \left(\Delta a_{1}^{\dagger} - \epsilon_{2} a_{2}\right) a_{2}^{\dagger} + \left(- \Delta a_{2}^{\dagger} - \epsilon_{1} a_{1}\right) a_{1}^{\dagger} + \left(\Delta a_{2}^{\dagger} - \epsilon_{3} a_{3}\right) a_{3}^{\dagger} + \left(- \Delta a_{3}^{\dagger} - \epsilon_{2} a_{2}\right) a_{2}^{\dagger} + \left(- \Delta a_{0} + \epsilon_{1} a_{1}^{\dagger}\right) a_{1} + \left(- \Delta a_{1} + \epsilon_{2} a_{2}^{\dagger}\right) a_{2} + \left(\Delta a_{1} + \epsilon_{0} a_{0}^{\dagger}\right) a_{0} + \left(- \Delta a_{2} + \epsilon_{3} a_{3}^{\dagger}\right) a_{3} + \left(\Delta a_{2} + \epsilon_{1} a_{1}^{\dagger}\right) a_{1} + \left(\Delta a_{3} + \epsilon_{2} a_{2}^{\dagger}\right) a_{2}$

```python
H = Dagger(c)*G*c
h1 = H[0]
h1
```

$\displaystyle \left(- \Delta a_{1}^{\dagger} - \epsilon_{0} a_{0}\right) a_{0}^{\dagger} + \left(\Delta a_{2}^{\dagger} - \epsilon_{3} a_{3}\right) a_{3}^{\dagger} + \left(\Delta a_{1} + \epsilon_{0} a_{0}^{\dagger}\right) a_{0} + \left(- \Delta a_{2} + \epsilon_{3} a_{3}^{\dagger}\right) a_{3} + \left(\Delta a_{0}^{\dagger} - \Delta a_{2}^{\dagger} - \epsilon_{1} a_{1}\right) a_{1}^{\dagger} + \left(\Delta a_{1}^{\dagger} - \Delta a_{3}^{\dagger} - \epsilon_{2} a_{2}\right) a_{2}^{\dagger} + \left(- \Delta a_{0} + \Delta a_{2} + \epsilon_{1} a_{1}^{\dagger}\right) a_{1} + \left(- \Delta a_{1} + \Delta a_{3} + \epsilon_{2} a_{2}^{\dagger}\right) a_{2}$

```python
h1.expand()
```

$\displaystyle \Delta a_{0}^{\dagger} a_{1}^{\dagger} - \Delta a_{1}^{\dagger} a_{0}^{\dagger} + \Delta a_{1}^{\dagger} a_{2}^{\dagger} - \Delta a_{2}^{\dagger} a_{1}^{\dagger} + \Delta a_{2}^{\dagger} a_{3}^{\dagger} - \Delta a_{3}^{\dagger} a_{2}^{\dagger} - \Delta a_{0} a_{1} + \Delta a_{1} a_{0} - \Delta a_{1} a_{2} + \Delta a_{2} a_{1} - \Delta a_{2} a_{3} + \Delta a_{3} a_{2} + \epsilon_{0} a_{0}^{\dagger} a_{0} - \epsilon_{0} a_{0} a_{0}^{\dagger} + \epsilon_{1} a_{1}^{\dagger} a_{1} - \epsilon_{1} a_{1} a_{1}^{\dagger} + \epsilon_{2} a_{2}^{\dagger} a_{2} - \epsilon_{2} a_{2} a_{2}^{\dagger} + \epsilon_{3} a_{3}^{\dagger} a_{3} - \epsilon_{3} a_{3} a_{3}^{\dagger}$

```python
result.expand()
```

$\displaystyle \Delta a_{0}^{\dagger} a_{1}^{\dagger} - \Delta a_{1}^{\dagger} a_{0}^{\dagger} + \Delta a_{1}^{\dagger} a_{2}^{\dagger} - \Delta a_{2}^{\dagger} a_{1}^{\dagger} + \Delta a_{2}^{\dagger} a_{3}^{\dagger} - \Delta a_{3}^{\dagger} a_{2}^{\dagger} - \Delta a_{0} a_{1} + \Delta a_{1} a_{0} - \Delta a_{1} a_{2} + \Delta a_{2} a_{1} - \Delta a_{2} a_{3} + \Delta a_{3} a_{2} + \epsilon_{0} a_{0}^{\dagger} a_{0} - \epsilon_{0} a_{0} a_{0}^{\dagger} + 2 \epsilon_{1} a_{1}^{\dagger} a_{1} - 2 \epsilon_{1} a_{1} a_{1}^{\dagger} + 2 \epsilon_{2} a_{2}^{\dagger} a_{2} - 2 \epsilon_{2} a_{2} a_{2}^{\dagger} + \epsilon_{3} a_{3}^{\dagger} a_{3} - \epsilon_{3} a_{3} a_{3}^{\dagger}$

The two are nearly identical. The center energy terms are bing counted twice...

```python
# When we remove the double counted energies teh two results are equal...
out = result.copy()
for i in [1, 2]:
    out += epsilon[i]*a[i]*Dagger(a[i]) - epsilon[i]*Dagger(a[i])*a[i]

h1.expand() == out.expand()
```

    True

# Programmetic construction of the BdG matrix
Considering the construction of the Bogoliubov de Gennes Matrix is procedural we should be able to construct it using some programmatic procedures.

The Hamiltonian that we will take here is

$\Delta \sum_{k=0}^{N} \big(c_{k+1}^{\dagger}c_{k}^{\dagger} + \mathrm{h.c.}\big)
- \mu \sum_{k=0}^{N} \big( c_{k}^{\dagger}c_{k} + \mathrm{h.c.} \big)
- \tau \sum_{k=0}^{N} \big(c_{k+1}^{\dagger}c_{k} + \mathrm{h.c.} \big)$

There is a conveniant way to construct these summations in Sympy using the Sum functions. This shows a nice mathematical representation of the sum until you actually execute the summation. Unfortunately the Indexed object are always commutation despite setting those to non-commutative. These makes it useles for operators.

```python
# Checking the commutative behavior of Indexed objects
Indexed.is_commutative = False
c = IndexedBase('c', commutative=False)
# This is not what we expect -> these commute
c[0]*c[1] + (c[1]*c[0])
```

$\displaystyle {c}_{0} {c}_{1} + {c}_{1} {c}_{0}$

```python
# This is what we are supposed to get
c = symbols('c:2', commutative=False)
c[0]*c[1] + c[1]*c[0]
```

$\displaystyle c_{0} c_{1} + c_{1} c_{0}$

Because of this we need to create a Sum function ourself. We will use this to construct the Hamiltonian

```python
# Simple summation function
def Sum(fnc, n, N):
    result = Add()
    for i in range(n+N):
        result += fnc(i)
    return result

# Create the non-commuting operators
n = 2
a = symbols(f"a_:{n}")
a = [Operator(ai) for ai in a]

# Construct the Hamiltonian
H0 = -Sum(lambda i: Dagger(a[i])*a[i] , 0, n) + \
     -tau* Sum(lambda i: Dagger(a[i+1])*a[i] + Dagger(a[i])*a[i+1] , 0, n-1) + \
     delta*Sum(lambda i: Dagger(a[i]*a[i+1]) + a[i+1]*a[i] , 0, n-1)

# NOTE: we already limit the index of summations that include i+1 terms to n-1
# Otherwise we would need include an extra operator term and later remove terms included that extra operator with
# H = drop_terms_containing(H0.expand(), [a[-1], Dagger(a[-1])])
# a = a[:-1]
H0
```

$\displaystyle \Delta \left(a_{1}^{\dagger} a_{0}^{\dagger} + a_{1} a_{0}\right) - \tau \left(a_{0}^{\dagger} a_{1} + a_{1}^{\dagger} a_{0}\right) - a_{0}^{\dagger} a_{0} - a_{1}^{\dagger} a_{1}$

So far we just created the original Hamiltonian. To construct a BdG Hamiltonian we need to use some commutation identities and rewrite the Hamiltonian.

The $\mu$-term

$a_{k}^{\dagger}a_{k} + a_{k+1}^{\dagger}a_{k} = 1 \to a_{k}^{\dagger}a_{k} = \frac{1}{2}a_{k}^{\dagger}a_{k} + \frac{1}{2}(1-a_{k}a_{k}^{\dagger})$

The $\Delta$-term

$a_{k+1}^{\dagger}a_{k}^{\dagger} + a_{k}^{\dagger}a_{k+1}^{\dagger} = 0 \to a_{k+1}^{\dagger}a_{k}^{\dagger} = \frac{1}{2}a_{k+1}^{\dagger}a_{k}^{\dagger} - \frac{1}{2}a_{k}^{\dagger}a_{k+1}^{\dagger}$

$a_{k+1} a_{k} + a_{k} a_{k+1} = 0 \to a_{k+1}a_{k} = \frac{1}{2}a_{k+1}a_{k}- \frac{1}{2}a_{k}a_{k+1}$

The $\tau$-term

$a_{k+1}^{\dagger}a_{k} + a_{k}^{\dagger}a_{k+1} = 0 \to a_{k+1}^{\dagger}a_{k} = \frac{1}{2}a_{k+1}^{\dagger}a_{k} - \frac{1}{2}a_{k}^{\dagger}a_{k+1}$

$a_{k}^{\dagger}a_{k+1} + a_{k+1}^{\dagger}a_{k} = 0 \to a_{k}^{\dagger}a_{k+1} = \frac{1}{2}a_{k}^{\dagger}a_{k+1} - \frac{1}{2}a_{k+1}^{\dagger}a_{k}$

After making these replacement we will be able to construct the BdG Hamiltonian

```python
# To replace them we need to trick because Sympy's subs method we do the replacement sequentially which means,
# parts that are added duning early replacements will we replaced later on during later replacements.
# The trick is to do the replcement is two stages using dummies in between.
submap = [[Dagger(a[i])*a[i], 1/2*(Dagger(a[i])*a[i] + 1 - a[i]*Dagger(a[i]))] for i in range(2)] + \
         [[Dagger(a[i]*a[j]), 1/2*(Dagger(a[j]*a[i]) - Dagger(a[i]*a[j]))] for i in range(2) for j in range(2)] + \
         [[a[i]*a[j], 1/2*(a[i]*a[j] - a[j]*a[i])] for i in range(2) for j in range(2)] + \
         [[Dagger(a[i+1])*a[i], 1/2*(Dagger(a[i+1])*a[i] - a[i]*Dagger(a[i+1]))] for i in range(2-1)] + \
         [[Dagger(a[i])*a[i+1], 1/2*(Dagger(a[i])*a[i+1] - a[i+1]*Dagger(a[i]))] for i in range(2-1)]

H = H0.expand()
for item in submap:
    key = Dummy()
    old, new = item

    H = H.subs(old, key)
    item[0] = key

H = H.subs(submap)
H.expand()
```

$\displaystyle 0.5 \Delta a_{0}^{\dagger} a_{1}^{\dagger} - 0.5 \Delta a_{1}^{\dagger} a_{0}^{\dagger} - 0.5 \Delta a_{0} a_{1} + 0.5 \Delta a_{1} a_{0} - 0.5 \tau a_{0}^{\dagger} a_{1} - 0.5 \tau a_{1}^{\dagger} a_{0} + 0.5 \tau a_{0} a_{1}^{\dagger} + 0.5 \tau a_{1} a_{0}^{\dagger} - 1.0 - 0.5 a_{0}^{\dagger} a_{0} - 0.5 a_{1}^{\dagger} a_{1} + 0.5 a_{0} a_{0}^{\dagger} + 0.5 a_{1} a_{1}^{\dagger}$

```python
# We construct the operator vectors!
c = Matrix(a + [Dagger(ai) for ai in a])
Dagger(c)
```

$\displaystyle \left[\begin{matrix}a_{0}^{\dagger} & a_{1}^{\dagger} & a_{0} & a_{1}\end{matrix}\right]$

```python
# Before we can create the
expr = ncollect(H.expand())
expr
```

$\displaystyle -1.0 + a_{0}^{\dagger} \left(0.5 \Delta a_{1}^{\dagger} - 0.5 \tau a_{1} - 0.5 a_{0}\right) + a_{1}^{\dagger} \left(- 0.5 \Delta a_{0}^{\dagger} - 0.5 \tau a_{0} - 0.5 a_{1}\right) + a_{0} \left(- 0.5 \Delta a_{1} + 0.5 \tau a_{1}^{\dagger} + 0.5 a_{0}^{\dagger}\right) + a_{1} \left(0.5 \Delta a_{0} + 0.5 \tau a_{0}^{\dagger} + 0.5 a_{1}^{\dagger}\right)$

```python
# This routine actually creates the BdG matrix Hamiltonian.
# First we express the Hamiltonian in terms of the leading (!!) operators.
expr = ncollect(H.expand())

# Next we collect the terms belonging to each of the elements in the operator vector Dagger(c).
# Those select the rows in the matrix
G = Matrix.zeros(len(c))
expr = ncollect(H.expand())
for i, l_ops in enumerate(Dagger(c)):
    h = get_coefficient(expr, l_ops)

    # For each term, we again match the operators but now to the operator vector c (!!)
    # This selects the columns.
    for j, r_ops in enumerate(c):
        coeff = get_coefficient(h, r_ops)
        G[i,j] = coeff

# This gives us the BdG Hamiltonian in matrix form
G
```

$\displaystyle \left[\begin{matrix}-0.5 & - 0.5 \tau & 0 & 0.5 \Delta\\- 0.5 \tau & -0.5 & - 0.5 \Delta & 0\\0 & - 0.5 \Delta & 0.5 & 0.5 \tau\\0.5 \Delta & 0 & 0.5 \tau & 0.5\end{matrix}\right]$

The original Hamiltonain in expanded sum-form can be obtained again (up to constant terms) using

$\mathrm{C^{\dagger} g C}$

where $\mathrm{C^{\dagger}} = [a_0^{\dagger}, a_1^{\dagger}, ... a_N^{\dagger}, a_0, a_1, ..., a_N]$.

```python
# Note the constant term '1' is missing
ncollect((Dagger(c)*G*c)[0].expand())
```

$\displaystyle a_{0}^{\dagger} \left(0.5 \Delta a_{1}^{\dagger} - 0.5 \tau a_{1} - 0.5 a_{0}\right) + a_{1}^{\dagger} \left(- 0.5 \Delta a_{0}^{\dagger} - 0.5 \tau a_{0} - 0.5 a_{1}\right) + a_{0} \left(- 0.5 \Delta a_{1} + 0.5 \tau a_{1}^{\dagger} + 0.5 a_{0}^{\dagger}\right) + a_{1} \left(0.5 \Delta a_{0} + 0.5 \tau a_{0}^{\dagger} + 0.5 a_{1}^{\dagger}\right)$

# alternative expression using Pauli matrices
Finally following https://topocondmat.org/w1_topointro/1D.html it should also be possible to express the BdG Hamiltonian using Pauli-matrixes. Ofcourse, this would help you actually constuct the Hamiltonian, it's just a nice way to write it.

```python
# We define the Pauli-matrices
sigma0 = Matrix([[1,0],[0,1]])
sigmax = Matrix([[0,1],[1,0]])
sigmay = Matrix([[0,-1j],[1j,0]])
sigmaz = Matrix([[1,0],[0,-1]])
```

The approach makes use of a 'selecting' vector $|n\rangle$ of size n consisting of a single non-zero element which is 1 i.e.

$|n\rangle = [..., 1,...]^{T}$ e.g. $|2\rangle = [0, 0, 1, ....]$

Note, we index from 0 here.

```python
# Conveniance function to create those special 'selecting vectors'
def nvec(N, k = 0):
    v = Matrix(N*[0])
    v[k] = 1
    return v
nvec(4, 1)
```

$\displaystyle \left[\begin{matrix}0\\1\\0\\0\end{matrix}\right]$

At the website they give the folowing identity:

$\mathrm{C^{\dagger}} \sigma_z |n\rangle \langle n| \mathrm{C} = 2 a_n^{\dagger} a_n - 1$

This seems weird because $\sigma_z$ is of size [2,2] but $|n\rangle \langle n|$ will be of size [n,n]. Moreover $\mathrm{C} is on size [2*n,1]$. To resolve this we need to take into account the Tensor product of $\sigma_z$ and $|n\rangle \langle n|$ which will be of size [2n, 2n]

$\mathrm{C^{\dagger}} \Big( \sigma_z \otimes |n\rangle \langle n| \Big) \mathrm{C} = 2 a_n^{\dagger} a_n - 1$

```python
# We check the identity
# We just reuse the c-vector of before
n0 = nvec(int(len(c)/2), 0)
eq = Dagger(c) * TensorProduct(mu*sigmaz, n0*Dagger(n0)) * c
eq[0]
```

$\displaystyle \mu a_{0}^{\dagger} a_{0} - \mu a_{0} a_{0}^{\dagger}$

Taking into account the commutation relation $a_0^{\dagger}a_0 + a_0 a_0^{\dagger} = 1$, this identity is indeed correct.

Next we can check the formation of the BdG Hamiltonian i.e.

```python
n1 = nvec(2, 1)
TensorProduct(tau*sigmaz + 1j*delta*sigmay, n0*Dagger(n1))
```

$\displaystyle \left[\begin{matrix}0 & \tau & 0 & 1.0 \Delta\\0 & 0 & 0 & 0\\0 & - 1.0 \Delta & 0 & - \tau\\0 & 0 & 0 & 0\end{matrix}\right]$

```python
Dagger(TensorProduct(tau*sigmaz + 1j*delta*sigmay, n0*Dagger(n1)))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\\tau^{\dagger} & 0 & - 1.0 \Delta^{\dagger} & 0\\0 & 0 & 0 & 0\\1.0 \Delta^{\dagger} & 0 & - \tau^{\dagger} & 0\end{matrix}\right]$

```python
# Lets construct the whole thing

result = Matrix.zeros(2*n, 2*n) #Add()
for ni in range(n):
    n0 = nvec(n, ni)
    result += -0.5*TensorProduct(mu*sigmaz, n0*Dagger(n0))

    if ni < n-1:
        n1 = nvec(n, ni+1)
        result += -0.5*TensorProduct(tau*sigmaz - 1j*delta*sigmay, n0*Dagger(n1))
        result += -0.5*TensorProduct(Dagger(tau*sigmaz - 1j*delta*sigmay), n1*Dagger(n0))
result
```

$\displaystyle \left[\begin{matrix}- 0.5 \mu & - 0.5 \tau & 0 & 0.5 \Delta\\- 0.5 \tau^{\dagger} & - 0.5 \mu & - 0.5 \Delta^{\dagger} & 0\\0 & - 0.5 \Delta & 0.5 \mu & 0.5 \tau\\0.5 \Delta^{\dagger} & 0 & 0.5 \tau^{\dagger} & 0.5 \mu\end{matrix}\right]$

```python
# The Hamiltonian in sum-form can again be obtained by multiplying with the operator vector.
ncollect((Dagger(c)*result*c)[0].expand())
```

$\displaystyle a_{0}^{\dagger} \left(- 0.5 \Delta a_{1}^{\dagger} - 0.5 \mu a_{0} - 0.5 \tau a_{1}\right) + a_{1}^{\dagger} \left(- 0.5 \mu a_{1} + 0.5 \Delta^{\dagger} a_{0}^{\dagger} - 0.5 \tau^{\dagger} a_{0}\right) + a_{0} \left(0.5 \Delta a_{1} + 0.5 \mu a_{0}^{\dagger} + 0.5 \tau a_{1}^{\dagger}\right) + a_{1} \left(0.5 \mu a_{1}^{\dagger} - 0.5 \Delta^{\dagger} a_{0} + 0.5 \tau^{\dagger} a_{0}^{\dagger}\right)$

```python

```


# **Cálculo Numérico - Avaliação 01**

# **Installing dependencies**

## **To install dependencies, run this cell**

```python
!pip install -r requirements.txt
```

## **Importing dependencies**

###  **To import dependencies, run this cell**

```python
from fractions import Fraction
from sympy import *
import numpy as np
import matplotlib.pyplot as plt
```

# **Exercício  00**

### Calculando matriz de Hilbert de ordem 4

```python
def Hilbert_matrix_string(a, b):
    return [[str(Fraction(1 / (i + j + 1)).limit_denominator()) for j in range(b)] for i in range(a)]

Hilbert_order_4_matrix = Hilbert_matrix_string(4,4)
print("Matriz de Hilbert ordem 4:")
Hilbert_order_4_matrix
```

### Calculando matriz inversa

```python
def Hilbert_matrix_array(a, b):
    return np.array([[Fraction(1 / (i + j + 1)).limit_denominator() for j in range(b)] for i in range(a)])

hilbert_matrix_array_order_4 = Hilbert_matrix_array(4,4).astype('float64')
Hilbert_inverse_matrix_array_order_4 = np.linalg.inv(hilbert_matrix_array_order_4)
print("\n\nMatriz inversa de Hilbert de ordem 4:")
Hilbert_inverse_matrix_array_order_4
```

### Descrevendo vetor solução usando pinv (pseudo-inverse) function

```python
pinv = np.linalg.pinv(hilbert_matrix_array_order_4)
b = [0 ,1, 10, 100]
solution_vector = pinv.dot(b)
print("O vetor solucao é dado por:")
solution_vector.tolist()
```

# **Exercício 01 parte a - Calculando xm para altura máxima**

```python
x = symbols('x')

g, y0, v0, m, c = 9.81, 100, 55, 80, 15
y = y0 + (m / c) * (v0 + ((m * g) / c)) * (1 - exp(- (c / m) * x)) - ((m * g) / c) * x
print('Equacao:')
y
```

### Altura maxima de xm é dada para y'(x) = 0

```python
y_diff = diff(y)
print("y(x)' = ", y_diff)
print("Calculando y(x)'= 0, resulta em xm, em metros, com o valor de")
solve(y_diff)[0]
```

# **Parte b - Calculando x para y(x) = 0**

```python
print("Valor de y(x) = 0: ")
solve(y, rational = False)[0]
```

### O processo deve ser iterativo, pois a equacao -52.32 * x + 672.373333333333 - 572.373333333333 * exp(-0.1875 * x) = 0 nao possui uma fórmula simples para resolucao da equacao.

```python
def Jacobian(v_str, f_list):
    vars = symbols(v_str)
    f = sympify(f_list)
    J = zeros(len(f),len(vars))
    for i, fi in enumerate(f):
        for j, s in enumerate(vars):
            J[i,j] = diff(fi, s)
    return J
```

# **Exercicio 02 parte a - Resolvendo o primeiro sistema de equacoes**

```python
x, y = symbols('x y')
f1 = x**2 + y**2
f2 = -exp(x) + y
print("Usando como chute inicial o par (I,I), pois a equacao nao possui solucoes reais, por conta de x² + y² = 0 possuir solucoes complexas, a solucao é dada por:")
nsolve((f1, f2), (x, y), (I, I))

```

```python
print("Matriz Jacobiana do sistema:")
Jacobian('x y',['x**2 + y**2', '-exp(x) + y'])
```

```python
x = [1,2,3,4]

y = [1, -1.33,-2.77,-3.233]

plt.plot(x, y)

plt.xlabel('Iteracoes')

plt.ylabel('Valores da raíz')

plt.title('Gráfico relacionando vetor solucao x X i')

plt.show()
```

# **Parte b - Mostrando que o segundo sistema de equacoes possui infinitas solucoes**

```python
print("Matriz Jacobiana do sistema:")
jacobian_matrix = Jacobian('x y', ['-exp(x) + y','-sin(x) + y'])
jacobian_matrix
```

```python
delta = 0.025
x, y = np.meshgrid(np.arange(-50, 4, delta),np.arange(-1, 1, delta))

plt.contour(x, y,- np.sin(x) + y, [0]
)
plt.contour(x, y, -np.exp(x) + y, [0])
plt.show()
```

### Como mostrado no gráfico, o sistema possui infinitas solucoes.

# **Exercicio 03 - descobrindo valor de d**

```python
d = symbols('d')
k1, k2, m, g, h = 40000, 40, 95, 9.81, 0.43
f = -(k1 * d + k2 * d ** (3/2))
energy_conservation = (2 * k2 * d ** (5/2))/5 + (1/2) * k1 * d ** 2 - m *g * d - m * g *h
print("Equacao:")
energy_conservation

```

```python
t = symbols('t')
f = (0.4 * t - t**3 + 0.141)
solved_equation = solve(f)
print("Os valores de t para que a matriz seja singular são:")
solved_equation
```

```python
print("Com a aproximacao inicial de d = 0.4, o valor de d, em metros, é igual a:")
nsolve(energy_conservation,0.4)
```


# Simple GP Regression

Gaussian processes are a powerful tool for nonlinear regression models.

Assume that we have **predictor** variables $\mathbf{X} = \{\mathbf{x_i}\}_{i=1}^N \in \mathbb{R}^d$ and **response** variables $\mathbf{y}=\{y_i \in \mathbb{R}\}_{i=1}^N$.

The response variables $\mathbf{y}$ are assumed to dependent on the predictors $\mathbf{X}$,

\begin{equation}
y_i \sim \mathcal{N}(f(\mathbf{x}_i),\sigma^2), \ i=1,\ldots,n,
\end{equation}

where $f$ is a mapping function. Treating $f$ as a random function, we assume that the distribution over $f$ is a **Gaussian process**,
$$
f \sim \mathcal{GP}(m(\mathbf{x}),k(\mathbf{x},\mathbf{x}')),
$$
where $m(\cdot)$ and $k(\cdot,\cdot)$ are the mean and kernel functions respectively.

## 1D regression example

We start by simulating some data

```julia
using GaussianProcesses
using Random

Random.seed!(20140430)
# Training data
n=10;                          #number of training points
x = 2π * rand(n);              #predictors
y = sin.(x) + 0.05*randn(n);   #regressors
```

The first step in modelling with Gaussian Processes is to choose mean functions and kernels which describe the process.

**Note** that all hyperparameters for the **mean** and **kernel** functions and $\sigma$ are given on the log scale. This is true for all strictly positive hyperparameters. Gaussian Processes are represented by objects of type 'GP' and constructed from observation data, a mean function and kernel, and optionally the amount of observation noise.

```julia
#Select mean and covariance function
mZero = MeanZero()                   #Zero mean function
kern = SE(0.0,0.0)                   #Sqaured exponential kernel (note that hyperparameters are on the log scale)

logObsNoise = -1.0                        # log standard deviation of observation noise (this is optional)
gp = GP(x,y,mZero,kern,logObsNoise)       #Fit the GP
```

    GP Exact object:
      Dim = 1
      Number of observations = 10
      Mean function:
        Type: MeanZero, Params: Float64[]
      Kernel:
        Type: SEIso{Float64}, Params: [0.0, 0.0]
      Input observations =
    [4.85461 5.17653 … 1.99412 3.45676]
      Output observations = [-0.967293, -1.00705, -1.0904, 0.881121, -0.333213, -0.976965, 0.915934, 0.736218, 0.950849, -0.306432]
      Variance of observation noise = 0.1353352832366127
      Marginal Log-Likelihood = -6.335

Once we've fit the `GP` function to the data, we can calculate the **predicted mean** and **variance** of the function at unobserved points $\{\mathbf{x}^\ast,y^\ast\}$, conditional on the observed data $\mathcal{D}=\{\mathbf{y},\mathbf{X}\}$. This is done with the `predict_y` function.

The `predict_y` function returns the mean vector $\mu(\mathbf{x}^\ast)$ and covariance matrix (variance vector if `full_cov=false`) $\Sigma(\mathbf{x}^\ast,\mathbf{x}^{\ast^\top})$ of the predictive distribution,
\begin{equation}
    y^\ast|\mathbf{x}^\ast,\mathcal{D} \sim \mathcal{N}(\mu(\mathbf{x}^\ast),\Sigma(\mathbf{x}^\ast,\mathbf{x}^{\ast^\top})+\sigma^2\mathbf{I}),
\end{equation}
where
\begin{eqnarray}
\mu(\mathbf{x}^\ast) &=& k(\mathbf{x}^\ast,\mathbf{X})(k(\mathbf{X}, \mathbf{X}) + \sigma_n^2 \mathbf{I})^{-1}\mathbf{y} \\ \ \mbox{and} \
\Sigma(\mathbf{x}^\ast,\mathbf{x}^{\ast}) &=& k(\mathbf{x}^\ast,\mathbf{x}^\ast) -k(\mathbf{x}^\ast,\mathbf{X})(k(\mathbf{X}, \mathbf{X})+ \sigma_n^2 \mathbf{I})^{-1} k(\mathbf{X},\mathbf{x}^\ast).
\end{eqnarray}

**Note** you can use the `predict_f` function to predict the latent function $\mathbf{f}^\ast$.

```julia
μ, σ² = predict_y(gp,range(0,stop=2π,length=100));
```

    ([0.357625, 0.384852, 0.412943, 0.441807, 0.471344, 0.501442, 0.53198, 0.562826, 0.593838, 0.624867  …  -0.669223, -0.63363, -0.597926, -0.562345, -0.527104, -0.492406, -0.458434, -0.425355, -0.393315, -0.362442], [0.603651, 0.557693, 0.512299, 0.468128, 0.425831, 0.386031, 0.349295, 0.316113, 0.286872, 0.261843  …  0.434056, 0.473396, 0.514594, 0.557168, 0.600593, 0.644326, 0.68782, 0.730548, 0.772021, 0.811799])

Plotting GPs is straightforward and utilises the recipes approach to plotting from the [Plots.jl](https://github.com/JuliaPlots/Plots.jl) package. More information about plotting GPs and the available functionality can be found in this [notebook](https://github.com/STOR-i/GaussianProcesses.jl/blob/master/notebooks/Plotting%20GPs.ipynb).

The default plot function `plot(gp)` outputs the predicted mean and variance of the function (i.e. uses `predict_f` in the background), with the uncertainty in the function represented by a confidence ribbon (set to 95% by default). All optional plotting arguments are given after `;`.

```julia
using Plots  #Load Plots.jl package

plot(gp; xlabel="x", ylabel="y", title="Gaussian process", legend=false, fmt=:png)      # Plot the GP
```

The hyperparameters are optimized using the [Optim.jl](https://github.com/JuliaOpt/Optim.jl) package. This offers users a range of optimization algorithms which can be applied to estimate the hyperparameters using type II maximum likelihood estimation. Gradients are available for all mean and kernel functions used in the package and therefore it is recommended that the user utilizes gradient based optimization techniques. As a default, the `optimize!` function uses the `L-BFGS` solver, however, alternative solvers can be applied.

```julia
using Optim
optimize!(gp; method=ConjugateGradient())   # Optimise the hyperparameters
```

    Results of Optimization Algorithm
     * Algorithm: Conjugate Gradient
     * Starting Point: [-1.0,0.0,0.0]
     * Minimizer: [-2.992856448832551,0.4636861230870647, ...]
     * Minimum: -3.275745e+00
     * Iterations: 27
     * Convergence: false
       * |x - x'| ≤ 0.0e+00: false
         |x - x'| = 4.62e-09
       * |f(x) - f(x')| ≤ 0.0e+00 |f(x)|: false
         |f(x) - f(x')| = -4.12e-14 |f(x)|
       * |g(x)| ≤ 1.0e-08: false
         |g(x)| = 5.69e-08
       * Stopped by an increasing objective: true
       * Reached Maximum Number of Iterations: false
     * Objective Calls: 66
     * Gradient Calls: 41

```julia
plot(gp; legend=false, fmt=:png)   #Plot the GP after the hyperparameters have been optimised
```

By default all hyperparameters or optimized. But the function `optimize!` allows also to force hyperparameters to remain constant or optimize them in a box.

```julia
optimize!(gp; kern = false)   # Don't optimize kernel hyperparameters
optimize!(gp; kernbounds = [[-1, -1], [1, 1]]) # Optimize the kernel parameters in a box with lower bounds [-1, -1] and upper bounds [1, 1]
```

    Results of Optimization Algorithm
     * Algorithm: Fminbox with L-BFGS
     * Starting Point: [-2.992856448832551,0.4636861230870647, ...]
     * Minimizer: [-2.992856448832551,0.4636861230870647, ...]
     * Minimum: -3.275745e+00
     * Iterations: 1
     * Convergence: true
       * |x - x'| ≤ 0.0e+00: true
         |x - x'| = 0.00e+00
       * |f(x) - f(x')| ≤ 0.0e+00 |f(x)|: true
         |f(x) - f(x')| = 0.00e+00 |f(x)|
       * |g(x)| ≤ 1.0e-08: false
         |g(x)| = 6.32e-08
       * Stopped by an increasing objective: true
       * Reached Maximum Number of Iterations: false
     * Objective Calls: 3
     * Gradient Calls: 3

MCMC, specifically a Hamiltonian Monte Carlo in this instance, can be run on the GPE hyperparameters through the `mcmc` function. Priors for hyperparameters of the mean and kernel parameters can be set through the `set_priors!` function. The log noise parameter of the GPE is a Uniform(0,1) distribution and currently can't be changed.

```julia
using Distributions

set_priors!(kern, [Normal(), Normal()]) # Uniform(0,1) distribution assumed by default if priors not specified
chain = mcmc(gp)
plot(chain', label=["Noise", "SE log length", "SE log scale"]; fmt=:png)
```

There is additional support for inference to be done in a fully Bayesian fashion through the use of an [elliptical slice sampler](http://proceedings.mlr.press/v9/murray10a/murray10a.pdf). While a HMC sampler can often be shown to be highly efficient, the sampler's efficiency can be highly dependent upon good initial choice of the sampler's hyperparameters. Conversely, ESS has no free parameters and is designed to be highly efficient in tightly correlated Gaussian posteriors: a geometry commonly found in Gaussian process models. Currently, inference via an ESS is only supported when the likelihood is Gaussian.

For advice on how to fit Gaussian processes with non-Gaussian data, see our notebook on [Poisson regression](https://github.com/STOR-i/GaussianProcesses.jl/blob/master/notebooks/Poisson%20regression.ipynb) or [classification](https://github.com/STOR-i/GaussianProcesses.jl/blob/master/notebooks/Classification.ipynb).

```julia
mhmc = mean(chain, dims=2)
mess = mean(ess_chain, dims=2)

```

    3×1 Array{Float64,2}:
     -2.444901016768212
      0.3721835201083439
     -0.10584635335452222

```julia
for (a, b) in zip(mhmc, mess)
    eq = abs(a-b)
    idx = max(abs(a), abs(b))
    return eq < idx/5
end
```

    true

```julia
mZero = MeanZero()                   #Zero mean function
kern = SE(0.0,0.0)                   #Sqaured exponential kernel (note that hyperparameters are on the log scale)
logObsNoise = -1.0                        # log standard deviation of observation noise (this is optional)

gpess = GP(x, y, mZero, kern, )       #Fit the GP

set_priors!(kern, [Normal(), Normal()]) # Uniform(0,1) distribution assumed by default if priors not specified
set_priors!(gpess.logNoise, [Distributions.Normal(-1.0, 1.0)])
ess_chain = ess(gpess)
plot(chain', label=["Noise", "SE log length", "SE log scale"]; fmt=:png)
```

## Multi-dimensional regression

The regression example above can be easily extended to higher dimensions. For the purpose of visualisation, and without loss of generality, we consider a 2 dimensional regression example.

```julia
#Training data
d, n = 2, 50;         #Dimension and number of observations
x = 2π * rand(d, n);                               #Predictors
y = vec(sin.(x[1,:]).*sin.(x[2,:])) + 0.05*rand(n);  #Responses
```

For problems of dimension>1 we can use isotropic (`Iso`) kernels or automatic relevance determination (`ARD`) kernels. For `Iso` kernels, the length scale parameter $\ell$ is the same for all dimensions. For `ARD` kernels, each dimension has different length scale parameter.

The `Iso` and `ARD` kernels are implemented automatically by replacing the single length scale parameter with a vector of parameters. For example, below we use the Matern 5/2 `ARD` kernel, if we wanted to use the `Iso` alternative then we would set the kernel as `kern=Mat(5/2,0.0,0.0)`.

In this example we use a composite kernel represented as the sum of a Matern 5/2 ARD kernel and a Squared Exponential isotropic kernel. This is easily implemented using the `+` symbol, or in the case of a product kernel, using the `*` symbol (i.e. `kern = Mat(5/2,[0.0,0.0],0.0) * SE(0.0,0.0)`).

```julia
mZero = MeanZero()                             # Zero mean function
kern = Matern(5/2,[0.0,0.0],0.0) + SE(0.0,0.0)    # Sum kernel with Matern 5/2 ARD kernel
                                               # with parameters [log(ℓ₁), log(ℓ₂)] = [0,0] and log(σ) = 0
                                               # and Squared Exponential Iso kernel with
                                               # parameters log(ℓ) = 0 and log(σ) = 0
```

    Type: SumKernel{Mat52Ard{Float64},SEIso{Float64}}
      Type: Mat52Ard{Float64}, Params: [-0.0, -0.0, 0.0]  Type: SEIso{Float64}, Params: [0.0, 0.0]

Fit the Gaussian process to the data using the prespecfied mean and covariance functions.

```julia
gp = GP(x,y,mZero,kern,-2.0)          # Fit the GP
```

    GP Exact object:
      Dim = 2
      Number of observations = 50
      Mean function:
        Type: MeanZero, Params: Float64[]
      Kernel:
        Type: SumKernel{Mat52Ard{Float64},SEIso{Float64}}
          Type: Mat52Ard{Float64}, Params: [-0.0, -0.0, 0.0]      Type: SEIso{Float64}, Params: [0.0, 0.0]
      Input observations =
    [3.05977 4.74752 … 2.82127 5.38224; 2.02102 4.27258 … 6.13114 1.56497]
      Output observations = [0.08509, 0.924505, 0.275745, -0.448035, -0.784758, -0.316803, -0.823483, -0.886726, 0.0059149, 0.414951  …  -0.413905, -0.347505, 0.46108, -0.204102, -0.538689, 0.554203, -0.874479, -0.0506017, -0.0215167, -0.745944]
      Variance of observation noise = 0.01831563888873418
      Marginal Log-Likelihood = -29.547

Using the [Optim](https://github.com/JuliaOpt/Optim.jl) package we have the option to choose from a range of optimize functions including conjugate gradients. It is also possible to fix the hyperparameters in either the mean function, kernel function or observation noise, by settting them to false in `optimize!` (e.g. `optimize!(...,domean=false)`).

```julia
optimize!(gp)                         # Optimize the hyperparameters
```

    Results of Optimization Algorithm
     * Algorithm: L-BFGS
     * Starting Point: [-2.0,-0.0,-0.0,0.0,0.0,0.0]
     * Minimizer: [-4.277211995773057,5.536664077544042, ...]
     * Minimum: -5.023830e+01
     * Iterations: 34
     * Convergence: false
       * |x - x'| ≤ 0.0e+00: false
         |x - x'| = 4.74e-02
       * |f(x) - f(x')| ≤ 0.0e+00 |f(x)|: false
         |f(x) - f(x')| = -1.36e-14 |f(x)|
       * |g(x)| ≤ 1.0e-08: false
         |g(x)| = 1.87e-05
       * Stopped by an increasing objective: true
       * Reached Maximum Number of Iterations: false
     * Objective Calls: 115
     * Gradient Calls: 115

A range of plotting options are availbe through the [Plots.jl](https://github.com/JuliaPlots/Plots.jl) package.

```julia
plot(contour(gp) ,heatmap(gp); fmt=:png)
```


# K Index Calculation

```python
import datetime as dt
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pooch
from viresclient import SwarmRequest
import ipywidgets as widgets

from warnings import filterwarnings
filterwarnings(action="ignore")

# Data dependencies (pooch caches this in ~/.cache/pooch/)
esk_k_ind_file = pooch.retrieve(
    "https://raw.githubusercontent.com/MagneticEarth/IAGA_SummerSchool2019/master/data/external/k_inds/esk/2003.esk",
    known_hash="233246e167a212cd1afa33ff2fe130fbc308cd2ae7971c6c2afcd363c9775c18"
)
```

## Calculating K-indices for a single observatory

The K-index is a local geomagnetic activity index devised by Julius Bartels in 1938 to give a simple measure of the degree of geomagnetic disturbance during each 3-hour (UT) interval seen at a single magnetic observatory. Data from the observatory magnetometers are used to assign a number in the range 0-9 to each 3-hour interval, with K=0 indicating very little geomagnetic activity and K=9 representing an extreme geomagnetic storm. The K-index was introduced at the time of photographic recording, when magnetograms recorded variations in the horizontal geomagnetic field elements declination (D) and horizontal intensity (H), and in the vertical intensity (Z).

To derive a K-index an observer would fit, __by eye__, a 'solar regular variation' ($S_R$) curve to the records of D and H and measure the range (maximum-minimum) of the deviation of the recording from the curve. The K-index was then assigned according to a conversion table, with the greatest range in D and H 'winning'. The north component (X) may be used instead of H, and the east component (Y) instead of D (X and Y will be used in the examples below and see http://isgi.unistra.fr/what_are_kindices.php for more details on the K-index). The vertical component Z is not used because it is liable to contamination by local induced currents.

The conversion from range in nanoteslas to index is quasi-logarithmic. The conversion table varies with latitude in an attempt to normalise the K-index distribution for observatories at different latitudes. The table for Eskdalemuir is shown below.

| K | 0 | 1 |2|3|4|5|6|7|8|9|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Lower bound (nT) | 0 | 8 |15|30|60|105|180|300|500|750|

This means that, for instance, K=2 if the measured range is in the interval \[15, 30\) nT.

There was a long debate in IAGA Division V about algorithms that could adequately reproduce the K-indices that an experienced observer would assign. The algorithms and code approved by IAGA are available at the International Service for Geomagnetic Indices: http://isgi.unistra.fr/softwares.php.

### Example

In the following cells, we **illustrate** a possible approach. We assume the so-called regular daily variation $S_R$ is made up of 24h, 12h and 8h signals (and, possibly, higher harmonics). A Fourier analysis can be used to investigate this. The functions in the cell below calculate Fourier coefficients from a data sample of one-minute data values over a 24-hour UT interval, and then synthesise a smooth fit using the Fourier coefficients.

For some days this simple approach to estimating $S_R$ seems to work well, on others it's obviously wrong. Think about another approach you might take.

We then attempt to calculate K-indices for the day chosen by computing the Fourier series up the the number of harmonics selected by subtracting the synthetic harmonic signal from the data, then calculating 3-hr ranges and converting these into the corresponding K-index. The functions to do are also included in the following cell.

```python
def fourier(v, nhar):
    npts    = len(v)
    f       = 2.0/npts
    t       = np.linspace(0, npts, npts, endpoint=False)*2*np.pi/npts
    vmn     = np.mean(v)
    v       = v - vmn
    cofs    = [0]*(nhar+1)
    cofs[0] = (vmn,0)
    for i in range(1,nhar+1):
        c, s    = np.cos(i*t), np.sin(i*t)
        cofs[i] = (np.dot(v,c)*f, np.dot(v,s)*f)
    return (cofs)

def fourier_synth(cofs, npts):
    nt  = len(cofs)
    syn = np.zeros(npts)
    t  = np.linspace(0, npts, npts, endpoint=False)*2*np.pi/npts
    for n in range(1, nt):
        for j in range(npts):
            syn[j] += cofs[n][0]*np.cos(n*t[j]) + cofs[n][1]*np.sin(n*t[j])
    return (syn)

# Define K-index conversion table for ESK
K_conversions = {
    f"K{level}": level_bin
    for level, level_bin in enumerate(
        (0, 8, 15, 30, 60, 105, 180, 300, 500, 750)
    )
}
# Define reverse mapping
nT_to_K = {v: k for k, v in K_conversions.items()}

def K_calc(d, synd, Kb=K_conversions):
    tmp = np.ptp((d-synd).reshape(8,180), axis=1)
    return(list(np.digitize(tmp, bins=list(Kb.values()), right=False)-1))

def load_official_K(filepath=esk_k_ind_file):
    df = pd.read_csv(filepath, skiprows=0, header=None, delim_whitespace=True,
                     parse_dates=[[2,1,0]], index_col=0)
    df = df.drop(3, axis=1)
    df.index.name='Date'
    df.columns = ['00','03','06','09','12','15','18','21']
    return(df)

def load_ESK_2003():
    request = SwarmRequest()
    request.set_collection(f"SW_OPER_AUX_OBSM2_:ESK", verbose=False)
    request.set_products(measurements=["B_NEC", "IAGA_code"])
    data = request.get_between(
        dt.datetime(2003, 1, 1),
        dt.datetime(2004, 1, 1),
    )
    df = data.as_dataframe(expand=True).drop(
        columns=["Spacecraft"]
    )
    df = df.rename(columns={f"B_NEC_{i}": j for i, j in zip("NEC", "XYZ")})
    return df
```

First, load in (X, Y, Z) one-minute data from Eskdalemuir for 2003 into a pandas dataframe.

```python
df_obs = load_ESK_2003()
df_obs.head()
```

Load the official K index data (available from <http://www.geomag.bgs.ac.uk/data_service/data/magnetic_indices/k_indices>) to compare with later.

```python
df_K_official = load_official_K()
df_K_official.head()
```

Evaluate K indices for a given day:
- For each of $X$ and $Y$:
- Perform a Fourier analysis on the data to find the regular daily variation, $S_R$
- Over each 3-hour interval, find the maximum differences from $S_R$
- Convert from nT to $K$ using the conversion table for ESK
- Pick the greater of $K(X)$ and $K(Y)$ and compare with the official K index

```python
def analyse_day(day=dt.date(2003, 1, 1), n_harmonics=3, df=df_obs, df_K_official=df_K_official):
    """Generate figure illustrating the K index calculation for a given day"""
    # Select given day
    _df = df.loc[day.isoformat()]
    _df_K = df_K_official.loc[day.isoformat()]
    # Select X & Y data and remove daily mean
    x = (_df["X"] - _df["X"].mean()).values
    y = (_df["Y"] - _df["Y"].mean()).values
    # Perform Fourier analysis of X & Y separately
    xcofs = fourier(x, n_harmonics)
    synx  = fourier_synth(xcofs, len(x))
    ycofs = fourier(y, n_harmonics)
    syny  = fourier_synth(ycofs, len(y))

    # Build plot
    t = np.linspace(0, 1440, 1440, endpoint=False)/60
    fig, axes = plt.subplots(2, 1, figsize=(15, 10), sharex=True)
    # Plot X & Y data with approximated variation
    axes[0].plot(t, x, color="tab:blue", alpha=0.5)
    axes[0].plot(t, synx, color="tab:blue", label="X")
    axes[0].plot(t, y, color="tab:red", alpha=0.5)
    axes[0].plot(t, syny, color="tab:red", label="Y")
    # Plot the differences
    axes[1].plot(t, (x-synx), color="tab:blue")
    axes[1].plot(t, (y-syny), color="tab:red")

    # Find and plot min/max bounds over 3-hourly intervals
    minX = np.min((x-synx).reshape(8, 180), axis=1)
    maxX = np.max((x-synx).reshape(8, 180), axis=1)
    minY = np.min((y-syny).reshape(8, 180), axis=1)
    maxY = np.max((y-syny).reshape(8, 180), axis=1)
    t_3hours = np.linspace(0, 1440, 9, endpoint=True)/60
    axes[1].fill_between(t_3hours, list(minX)+[0], list(maxX)+[0], step="post", color="tab:blue", alpha=0.5)
    axes[1].fill_between(t_3hours, list(minY)+[0], list(maxY)+[0], step="post", color="tab:red", alpha=0.5)

    # Determine K index from each of X & Y
    K_X = np.digitize((maxX-minX), bins=list(K_conversions.values()), right=False) - 1
    K_Y = np.digitize((maxY-minY), bins=list(K_conversions.values()), right=False) - 1
    for i, (K_X_i, K_Y_i) in enumerate(zip(K_X, K_Y)):
        # Display determined K from X & Y
        px = i*3
        py = axes[1].get_ylim()[1]
        axes[1].annotate(
            f"K(X): {K_X_i}", (px, py), xytext=(30, 18),
            textcoords="offset pixels", color="tab:blue", size=12,
        )
        axes[1].annotate(
            f"K(Y): {K_Y_i}", (px, py), xytext=(30, 3),
            textcoords="offset pixels", color="tab:red", size=12,
        )
        # Display comparison with the official K index
        K_ours = max(K_X_i, K_Y_i)
        K_official = _df_K[i]
        axes[1].annotate(
            f"{K_ours}\n{K_official}",
            (i*3, axes[1].get_ylim()[0]), xytext=(40, -70), textcoords="offset pixels"
        )
    axes[1].annotate(
        f"Determined K:\nOfficial K:",
        (0, axes[1].get_ylim()[0]), xytext=(-80, -70), textcoords="offset pixels"
    )

    # Finalise figure
    for ax in axes:
        ax.grid()
        ax.xaxis.set_ticks(np.arange(0, 27, 3))
    axes[1].set_ylabel("Residuals [nT]")
    axes[1].set_xlabel("UT [hour]")
    axes[0].set_ylabel("[nT]")
    axes[0].legend(loc="upper right")
    fig.suptitle(f"ESK: {day.isoformat()}", y=0.9)

    return fig, axes

def make_widgets_K_index_calc():
    day = widgets.SelectionSlider(
        options=[t.date() for t in pd.date_range(dt.date(2003, 1, 1), dt.date(2003, 12, 31))],
        description="Select day:", layout=widgets.Layout(width='700px')
    )
#     day = widgets.DatePicker(value=dt.date(2003, 1, 1), description="Select day:")
    n_harmonics = widgets.SelectionSlider(options=range(1, 11), value=3, description="# harmonics:")
    return widgets.VBox(
        [day,
         n_harmonics,
         widgets.interactive_output(
             analyse_day,
             {"day": day, "n_harmonics": n_harmonics}
         )],
    )

make_widgets_K_index_calc()
```

## Statistics of the K index

We will use the official K index from ESK to probe some statistics through the year 2003.

Histograms of the K indices for each 3-hour period:

```python
axes = df_K_official.hist(
    figsize=(12, 12), bins=range(11), sharey=True, align="left", rwidth=0.8,
)
plt.suptitle('ESK 2003: Distribution of K-indices for each 3-hour interval')
axes[-1, 0].set_ylabel("Frequency")
axes[-1, 0].set_xlabel("K");
```

... plotted side by side:

```python
plt.figure(figsize=(7,7))
plt.hist(df_K_official.values, bins=range(11), align='left')
plt.legend(df_K_official.columns)
plt.ylabel('Number of 3-hour intervals')
plt.xlabel('K');
```

... and stacked together:

```python
plt.figure(figsize=(7,7))
plt.hist(df_K_official.values, bins=range(11), stacked=True, align='left', rwidth=0.8)
plt.legend(df_K_official.columns)
plt.ylabel('Number of 3-hour intervals')
plt.xlabel('K');
```

We also compute a daily sum of the K-indices for the 2003 file, and list days with high and low summed values. Note that this summation is not really appropriate because the K-index is quasi-logarithmic, however, this is a common simple measure of quiet and disturbed days. (These might be interesting days for you to look at.)

```python
df_K_official['Ksum'] = df_K_official.sum(axis=1)
Ksort = df_K_official.sort_values('Ksum')
print('Quiet days: \n\n', Ksort.head(10), '\n\n')
print('Disturbed days: \n\n', Ksort.tail(10))
```

## Note on the Fast Fourier Transform

In the examples above we computed Fourier coefficients in the 'traditional' way, so that if $F(t)$ is a Fourier series representation of $f(t)$, then,

$$
\begin{align}
F(t) &= A_o+\sum_{n=1}^N A_n \cos\left(\frac{2\pi nt}{T}\right)+B_n \sin\left(\frac{2\pi nt}{T}\right)
\end{align}
$$

where $T$ is the fundamental period of $F(t)$. The $A_n$ and $B_n$ are estimated by

$$
\begin{align}
A_o&=\frac{1}{T}\int_0^T f(t) dt\\
A_n&=\frac{2}{T}\int_0^T f(t)\cos\left(\frac{2\pi nt}{T}\right) dt\\
B_n&=\frac{2}{T}\int_0^T f(t)\sin\left(\frac{2\pi nt}{T}\right) dt
\end{align}
$$

With $N$ samples of digital data, the integral for $A_n$ may be replaced by the summation

$$
\begin{align}
A_n&=\frac{2}{T}\sum_{j=0}^{N-1} f_j\cos\left(\frac{2\pi nj\Delta t}{T}\right) \Delta t\\
&=\frac{2}{N}\sum_{j=0}^{N-1} f_j\cos\left(\frac{2\pi nj}{N}\right)
\end{align}
$$

where the sampling interval $\Delta t$ is given by $T = N \Delta t$ and $f_j = f(j \Delta t)$. A similar expression applies for the $B_n$, and these are the coefficients returned by the function _fourier_ above.

The fast Fourier transform (FFT) offers a computationally efficient means of finding the Fourier coefficients. The conventions for the FFT and its inverse (IFFT) vary from package to package. In the _scipy.fftpack_ package, the FFT of a sequence $x_n$ of length $N$ is defined as

$$
\begin{align}
y_k&=\sum_{n=0}^{N-1} x_n\exp\left(-\frac{2\pi i\thinspace kn}{N}\right)\\
&=\sum_{n=0}^{N-1} x_n\left(\cos\left(\frac{2\pi \thinspace kn}{N}\right)-i\sin\left(\frac{2\pi \thinspace kn}{N}\right)\right)
\end{align}
$$

with the inverse defined as,

$$
\begin{align}
x_n&=\frac{1}{N}\sum_{k=0}^{N-1} y_k\exp\left(\frac{2\pi i\thinspace kn}{N}\right)\\
\end{align}
$$

(The _scipy_ documentation is perhaps a little confusing here because it explains the order of the $y_n$ as being $y_1,y_2, \dots y_{N/2-1}$ as corresponding to increasing positive frequency and $y_{N/2}, y_{N/2+1}, \dots y_{N-1}$ as ordered by decreasing negative frequency, for $N$ even. See: https://docs.scipy.org/doc/scipy/reference/tutorial/fft.html.)

The interpretation is that if $y_k=a_k+ib_k$ then will have (for $N$ even), $y_{N-k} = a_k-ib_k$ and so

$$
\begin{align}
a_k&=\frac{1}{2}\text{Re}\left(y_k+y_{N-k}\right)\\
b_k&=\frac{1}{2}\text{Im}\left(y_k-y_{N-k}\right)
\end{align}
$$

and so we expect the relationship to the digitised Fourier series coefficients returned by the function _fourier_ defined above to be,

$$
\begin{align}
A_k&=\phantom{-}\frac{1}{N}\text{Re}\left(a_k+a_{N-k}\right)\\
B_k&=-\frac{1}{N}\text{Im}\left(b_k-b_{N-k}\right)
\end{align}
$$

The following shows the equivalence between the conventional Fourier series approach and the FFT.

```python
from scipy.fftpack import fft

# Compute the fourier series as before
_df = df_obs.loc["2003-01-01"]
x = (_df["X"] - _df["X"].mean()).values
xcofs = fourier(x, 3)
# Compute using scipy FFT
npts = len(x)
xfft = fft(x)

# Compare results for the 24-hour component
k    = 1
print('Fourier coefficients: \n', f'A1 = {xcofs[1][0]} \n', f'B1 = {xcofs[1][1]} \n')
print('scipy FFT outputs: \n', f'a1 = {np.real(xfft[k]+xfft[npts-k])/npts} \n', \
      f'b1 = {-np.imag(xfft[k]-xfft[npts-k])/npts} \n')
```

## References

Menvielle, M. et al. (1995) ‘Computer production of K indices: review and comparison of methods’, Geophysical Journal International. Oxford University Press, 123(3), pp. 866–886. doi: [10.1111/j.1365-246X.1995.tb06895.x](https://doi.org/10.1111/j.1365-246X.1995.tb06895.x).


### Scalar field cosmology

In this notebook, we introduce the simplest scalar field cosmology, dubbed $quintessence$. In this simple example, the Universe's expansion (represented by a nonvanishing Hubble parameter $H$) is sourced only by a scalar field $\phi$, a theoretical device that we typically use to study dark energy.

The Friedmann equation is then
\begin{equation}
3 M_\text{pl}^2 H^2 = V(\phi) + \frac{\dot{\phi}^2}{2} .
\end{equation}
On the other hand, the equation of motion for the scalar field is
\begin{equation}
\ddot{\phi} + 3 H \dot{\phi} + V' \left( \phi \right) = 0 .
\end{equation}
Incidentally, this looks just like the equation of motion for a ball, with coordinate $x$, rolling in a conservative potential $V(x)$ with a time dependent friction force $-b(t) \dot{x}$. For the rest of this notebook, we consider only the quadratic potential
\begin{equation}
V(\phi) = \frac{m}{2} \phi^2 ,
\end{equation}
which could stand for an inflaton of mass $m$. Admittedly, this is far from the most realistic model, but it gets things done for a student.

So, we now integrate the field equations (those two equations above). To do this, we eliminate $H$ from scalar field equation using the . In doing this, we obtain
\begin{equation}
\ddot{\phi} + \frac{\sqrt{3}}{M_\text{pl}} \left( V(\phi) + \frac{\dot{\phi}^2}{2} \right)^{1/2} \dot{\phi} + V' \left( \phi \right) = 0 .
\end{equation}
We then solve this first and then put the result into the Friedmann equation to obtain the scale factor $a(t)$. Then, we use the solution to integrate the Friedmann equation.

```python
import numpy as np
from pylab import *
from scipy.integrate import odeint

# we set Planck mass to unity

x_initial = np.array([1., 0.])
t_space = np.linspace(0., 50., 1000)

def f(x, t):
    '''the scalar field ode'''
    return np.array([ x[1] , -np.sqrt(3)*np.sqrt( 0.5*(x[0]**2.) + 0.5*(x[1]**2.) )*x[1] - x[0] ])

inflaton = odeint(f, x_initial, t_space)

phi = inflaton[:, 0]
phi_prime = inflaton[:, 1]

# plots inflaton field and its derivative

rc('xtick', labelsize = 20)
rc('ytick', labelsize = 20)

plot(t_space, phi, 'r-', linewidth = 3.0, label = r'$\phi$')
plot(t_space, phi_prime, 'b--', linewidth = 3.0, label = r'$\dot{\phi}$')
xlim(min(t_space), max(t_space))
xlabel(r'$t$', fontsize = 20)
ylabel(r'$\phi,\dot{\phi}$', fontsize = 20)
legend(loc = 'upper right', prop = {'size': 15})
show()
```

Thus, we see that the field $\phi$ would oscillate simply about the equilibrium of the potential $V(\phi)$, just as a rolling ball, while friction throws away its energy, thus, the decreasing amplitude.

The expansion history sourced by this rolling field $\phi$ is obtained as follows. We construct a function $V\left( \phi \right) + \dot{\phi}^2/2$ out of the inflaton solution and then integrate the Friedmann equation for the scale factor, noting that $H = \dot{a}/a$.

```python
def H_squared(t):
    '''Friedmann equation right hand side
    the input t must be within the range of integration'''
    abs_difference = abs( t - t_space )
    ind = list(abs_difference).index(min(abs_difference))

    return 0.5*(phi[ind]**2.) + 0.5*(phi_prime[ind]**2.)

def g(y, t):
    '''ode for the scale factor'''
    return np.array([ np.sqrt(H_squared(t)/3)*y[0] ])

a_initial = np.array([0.01]) # initial condition on scale factor at t = 0

scale_factor = odeint(g, a_initial, t_space)

a = scale_factor[:, 0]

# plots solution

H_sq_list = []
for each in t_space:
    H_sq_list.append(H_squared(each)/3.)
H_sq_array = np.array(H_sq_list) # construct array version of H^2

plot(t_space, a, 'r-', linewidth = 3.0, label = r'$a$')
plot(t_space, H_sq_array, 'b--', linewidth = 3.0, label = r'$H^2$')
xlim(min(t_space), max(t_space))
xlabel(r'$t$', fontsize = 20)
ylabel(r'$a,H^2$', fontsize = 20)
legend(loc = 'upper right', prop = {'size': 15})
show()
```

This shows us that the field $\phi$ would indeed source an expanding Universe, but it can't continue this work by itself. The expansion would slow down or decelerate, $\ddot{a} < 0$. In practice, a bunch of cosmic ingredients (radiation, baryons, dark matter, dark energy) should help out to support the observed expansion history.


# DATA 5600: Introduction to Regression and Machine Learning for Analytics

## __Chapter 3: Some Basic Methods in Mathematics and Probability__

Author:      Tyler J. Brough

Updated: September 15, 2021

---

## __Section 3.0: Python Imports__

```python
import numpy as np
from scipy import stats
import seaborn as sns
import matplotlib.pyplot as plt

sns.set_style('darkgrid')
plt.rcParams['figure.figsize'] = [10, 5]
```

---

## __Section 3.1: Weighted Averages__

__Example:__ Populations and average ages of countries in North America

| Stratum, $j$ | Label         | Population, $N_{j}$ | Average age, $\bar{y}_{j}$ |
|:------------:|:--------------|--------------------:|:--------------------------:|
| 1            | United States | 310 million         | 36.8                       |
| 2            | Mexico        | 112 million         | 26.7                       |
| 3            | Canada        | 34 million          | 40.7                       |

* In 2010 there 456 million people in North America
    - 310 million in the United States
    - 112 million in Mexico
    - 34 million in Canada

The average age of all North Americans is a ___weighted average___:

$$
\large{\mbox{average age} = \frac{310,000,000 \ast 36.8 + 112,000,000 \ast 26.7 + 34,000,000 \ast 40.7}{310,000,000 + 112,000,000 + 34,000,000} = 34.6 \mbox{ years.}}
$$

* The total poplation was $310 + 112 + 34 = 456$ million

* We can rewrite the above as:

$$
\begin{align}
\mbox{average age} &= \frac{310}{456} \ast 36.8 + \frac{112}{456} \ast 26.7 + \frac{34}{456} \\
                   & \\
                   &= 0.6798 \ast 36.8 + 0.2456 \ast 26.7 + 0.0746 \ast 40.7 \\
                   & \\
                   & = 34.6
\end{align}
$$

* The proportions $0.6798$, $0.2456$, and $0.0746$ are the _weights_ of the countries in this weighted average.

We can equivalently write a weighted average in summation notation:

$$
\large{\mbox{weighted average} = \frac{\sum_{j} N_{j} \bar{y}_{j}}{\sum_{j} N_{j}}}
$$

* where $j$ indexes countries and the sum adds over all the _strata_ (in this case, the three countries)

__NB:__ we can also write a weighted average as: $\mbox{weighted average} = \sum_{j} \omega_{j} \bar{y}_{j}$ where $\omega_{j} \quad$ are the weights and $\bar{y}_{j}\quad$ are the strata values.

* So in our example above: $\omega_{1} = 0.6798$, $\omega_{2} = 0.2456$, and $\omega_{3} = 0.0746$

* $\bar{y}_{1} = 36.8$, $\bar{y}_{2} = 26.7$, and $\bar{y}_{3} = 40.7$

* So then:

$$
\begin{align}
\mbox{weighted average} &= \sum_{j} \omega_{j} \bar{y}_{j} \\
                        &= (\omega_{1} \ast \bar{y}_{1}) + (\omega_{2} \ast \bar{y}_{2}) + (\omega_{3} \ast \bar{y}_{3}) \\
                        &= (0.6798 \ast 36.8) + (0.2456 \ast 26.7) + (0.0746 \ast 40.7) \\
                        &= 34.6
\end{align}
$$

```python
## It's trivial to do weighted averages in Python
populations = np.array([310000000, 112000000, 34000000])
average_ages = np.array([36.8, 26.7, 40.7])
weights = populations / populations.sum()
weighted_average_age = np.sum(weights * average_ages) # NB: the multiplication is element-wise

## Print a nicely formatted string
print(f"\nThe weighted average age in North America is: {weighted_average_age : 0.2f} years\n")
```

    The weighted average age in North America is:  34.61 years

---

## __Section 3.2: Vectors and Matrices__

* A list of numbers is called a ___vector___

* A rectangular array of numbers is called a ___matrix___

* Vectors and matrices are useful in regression to represent predictions for many use cases using a single model

__Example:__ Model for predicting incumbent party's vote percentage in US presidential elections from economic conditions the years preceding

$$
\mbox{Predicted vote share} = 46.3 + 3.0 \ast (\mbox{growth rate of average personal income})
$$

Which we can write as:

$$
\begin{align}
\hat{y} &= 46.3 + 3.0x \\
        & \\
\hat{y} &= \hat{a} + \hat{b} x \\
\end{align}
$$

* The expressions $\hat{a}$ and $\hat{b}$ denote estimates - the coefficients: $\hat{a} = 46.3$ and $\hat{b} = 3.0$

* These were obtained by fitting a line to past data

* $\hat{y}$ denotes a predicted value

* We will use $y$ to represent an actual election result, and $\hat{y}$

Let's apply this model to a few different cases:

1. $x = -1$. A rate of growth of $-1\%$ (that is, a $1\%$ decline in the economy) translates into an incumbent party vote share of $46.3 + 3.0 \ast (-1) = 43.3\%$

2. $x = 0$. If there is zero economic growth in the year preceding the presidential election, the model predicts that the incumbent party's candidate will receive
   $46.3 + 3.0 \ast 0 = 46.3\%$ of the two-party vote; that is, he or she is predicted to lose the election.

3. $x = 3$. A $3\%$ rate of economic growth tranlsates to the incumbent party's candidate winning $46.3 + 3.0 \ast 3 = 55.3\%$ of the vote.

We can put these three predictions together:

$$
\begin{align}
\hat{y}_{1} & = 43.3 = 46.3 + 3.0 \ast (-1) \mbox{,} \\
            &  \\
\hat{y}_{2} & = 46.3 = 46.3 + 3.0 \ast (0) \mbox{,} \\
            & \\
\hat{y}_{3} & = 55.3 = 46.3 + 3.0 \ast (3) \mbox{,}
\end{align}
$$

And we can write them as vectors:

$$
\hat{y} = \begin{bmatrix} 43.3 \\ 46.3 \\ 55.3 \\ \end{bmatrix} = \begin{pmatrix} 46.3 + 3.0 \ast (-1) \\ 46.3 + 3.0 \ast (0)  \\ 46.3 + 3.0 \ast (3)  \\ \end{pmatrix}
$$

Or in matrix form:

$$
\hat{y} = \begin{pmatrix} 43.3 \\ 46.3 \\ 55.3 \\ \end{pmatrix} = \begin{pmatrix} 1 \quad -1 \\ 1 \quad 0 \\ 1 \quad 3 \\ \end{pmatrix} \begin{pmatrix} 46.3 \\ 3.0 \end{pmatrix}
$$

Or even more abstractly:

$$
\hat{y} = X \hat{\beta}
$$

* Here $y$ and $x$ are vectors of length 3

* $X$ is a $3 \times 2$ matrix with a column of ones and a column equal to the vector $x$

* $\hat{\beta} = (46.3, 3.0)$ is a estimated coefficients

```python
## We can represent these with ndarrays from numpy
X = np.array([[1,-1], [1, 0], [1, 3]])
X
```

    array([[ 1, -1],
           [ 1,  0],
           [ 1,  3]])

```python
beta_hat = np.array([46.3, 3.0])
beta_hat
```

    array([46.3,  3. ])

```python
y_hat = np.dot(X, beta_hat)
y_hat
```

    array([43.3, 46.3, 55.3])

```python

```

---

## __Section 3.3: Graphing a Line__

* To use linear regression effectively, you need to understand the algebra and geometry of straight lines

$$
\Large{y = a + bx}
$$

* $a$ is the _intercept_, the value of $y$ when $x = 0$

* $b$ is the _slope_ of the line
    - The line slopes upward if $b > 0$
    - The line slopes downward if $b < 0$
    - The line is horizontal if $b = 0$

* The larger the $b$ is in absolute value the steeper the line will be

* Figure 3.2 shows the line $y = 1007 - 0.39x$

* $y = 1007$ when $x = 0$

* $y$ decreases by $0.39$ when $x$ increases by 1 unit

* This line approximates the trajectory of the world record time (in seconds) for the mile run from 1900 to 2000

This is easy in `Python` with the `matplotlib` module

```python
## Plot a simple line in matplotlib.pyplot
x = np.linspace(0, 2, 100)
y = 1007 - 0.393*x
plt.plot(x, y, lw = 3.0, color='blue', alpha=0.75)
plt.show()
```

```python
np.arange??
```

    Docstring:
    arange([start,] stop[, step,], dtype=None, *, like=None)

    Return evenly spaced values within a given interval.

    Values are generated within the half-open interval ``[start, stop)``
    (in other words, the interval including `start` but excluding `stop`).
    For integer arguments the function is equivalent to the Python built-in
    `range` function, but returns an ndarray rather than a list.

    When using a non-integer step, such as 0.1, the results will often not
    be consistent.  It is better to use `numpy.linspace` for these cases.

    Parameters
    ----------
    start : integer or real, optional
        Start of interval.  The interval includes this value.  The default
        start value is 0.
    stop : integer or real
        End of interval.  The interval does not include this value, except
        in some cases where `step` is not an integer and floating point
        round-off affects the length of `out`.
    step : integer or real, optional
        Spacing between values.  For any output `out`, this is the distance
        between two adjacent values, ``out[i+1] - out[i]``.  The default
        step size is 1.  If `step` is specified as a position argument,
        `start` must also be given.
    dtype : dtype
        The type of the output array.  If `dtype` is not given, infer the data
        type from the other input arguments.
    like : array_like
        Reference object to allow the creation of arrays which are not
        NumPy arrays. If an array-like passed in as ``like`` supports
        the ``__array_function__`` protocol, the result will be defined
        by it. In this case, it ensures the creation of an array object
        compatible with that passed in via this argument.

        .. note::
            The ``like`` keyword is an experimental feature pending on
            acceptance of :ref:`NEP 35 <NEP35>`.

        .. versionadded:: 1.20.0

    Returns
    -------
    arange : ndarray
        Array of evenly spaced values.

        For floating point arguments, the length of the result is
        ``ceil((stop - start)/step)``.  Because of floating point overflow,
        this rule may result in the last element of `out` being greater
        than `stop`.

    See Also
    --------
    numpy.linspace : Evenly spaced numbers with careful handling of endpoints.
    numpy.ogrid: Arrays of evenly spaced numbers in N-dimensions.
    numpy.mgrid: Grid-shaped arrays of evenly spaced numbers in N-dimensions.

    Examples
    --------
    >>> np.arange(3)
    array([0, 1, 2])
    >>> np.arange(3.0)
    array([ 0.,  1.,  2.])
    >>> np.arange(3,7)
    array([3, 4, 5, 6])
    >>> np.arange(3,7,2)
    array([3, 5])
    Type:      builtin_function_or_method

---

## __Section 3.4: Exponential and Power-Law Growth and Decline; Logarthmic and Log-Log Relationships__

* We can express a more general class of relationships be allowing logarithmic transformations

* The formula $\log{(y)} = a + bx$ represents exponential growth (if $b > 0$) or decline (if $b < 0$): $y = Ae^{bx}$ where $A = e^{a}$

* The parameter $A$ is the value of $y$ when $x = 0$

* The parameter $b$ determines the rate of growth or decline

* A one-unit difference in $x$ corresponds to an additive difference of $b$ in $\log{(y)}$
    - Thus a multiplicative factor of $e^{b}$ in $y$

#### __Exponential Growth__

__Two Examples:__

___1. Exponential growth___

- Suppose the world population starts at 1.5 billion in the year 1900 and increases exponentially,, doubling every 50 years (not an accurate description, just a crude approximation)

- We can write this as $y = A \ast 2^{(x - 1900)/50}$

- $A = 1.5 \ast 10^{9}$

- Equivalently, $y = A e^{\log{(2)}/50(x - 1900)} = A e^{0.014(x - 1900)}$

- NB: in statistics we use "log" to refer to the natural logarithm (log base $e$, not base $10$). See Section 12.4

- The model $y = A e^{0.014(x - 1900)}$ is exponential growth with a rate of $0.014$, which implies that $y$ increases by a factor of $e^{0.014}$ = 1.014$ per year

- Or $e^{0.14} = 1.15$ per ten years

- Or $e^{1.4} = 4.0$ per hundred years

- We can take the log of both sides of the equation to get

$$
\large{\log{(y)} = 21.1 + 0.014(x - 1900)}
$$

- NB: $\log{(A)} = \log{(1.5 \ast 10^{9}} = 21.1$

___2. Exponential Decline___

- Consider an asset that is initially worth $\$1000$ and declines in value by $20\%$ each year

- Then its value at year $x$ will be $y = 1000 \ast 0.8^{x}$

- Or $y = 1000 e^{\log{(0.8)}x} = 1000e^{-0.22x}$

- Logging both sides yields the following

$$
\large{\log{(y)} = \log{(1000)} - 0.22x = 6.9 - 0.22x}
$$

#### __Power-Law Growth and Decline__

- The formula $\log{(y)} = a + b \log{(x)}$ represents power-law growth (if $b > 0$) or decline (if $b < 0$):

- $y = Ax^{b}$, where $A = e^{a}$

- The parameter $A$ is the value of $y$ when $x = 1$

- The parameter $b$ determines the rate of growth or decline

- A one-unit difference in $\log{(x)}$ corresponds to an additive difference of $b$ in $\log{(y)}$

__Two Examples:__

__1. Power Law__

- Let $y$ be the area of a square and $x$ be its perimeter

- Then $y = (x/4)^{2}$

- Take the log of both sides: $\log{(y)} = 2 (\log{(x)} - \log{(4)}) = -2.8 + 2 \log{(x)}$

__2. Non-integer Power Law__

- Let $y$ be the surface area of a cube and $x$ be its volume

- If $L$ is the length of a side of the cube, then $y = 6L^{2}$ and $x = L^{3}$,

- Hence the relation between $y$ and $x$ is $y = 6x^{2/3}$

- Thus $\log{(y)} = \log{(6)} + \frac{2}{3} \log{(x)} = 1.8 + \frac{2}{3}\log{(x)}$

#### __Interpreting Power-Law  or Log-Log Regressions__

- Placeholder item 1

- Placeholder item 2

- ...

---

## __Section 3.5: Probability Distributions__

- In section 3.3 we reviewed straight-line prediction which is the deterministic part of linear regression (and a key building block)

- Now we want to introduce probability distributions and random variables (we need these because our models do not fit our data exactly)

- Probability distributions represent the unmodeled aspects of reality - the ___error term $\epsilon$___ in the expression

$$
\Large{y = a + b x + \epsilon}
$$

- When a ball is drawn "at random", the ___random variable___ is what is written on the ball

- This treatment is rather informal (not axiomatic)

- We mix conceptual definitions with mathematical formulas where we think these will be useful

#### __Application Areas of Probability Distributions in Regression Modeling__

- Distributions of data (for example, heights of men, incomes of women, political party preference), for which we will use the notation $y_{i}, \mbox{ for } i = 1, \ldots, n$.

- Distributions of error terms, which we will write as $\epsilon_{i}, \mbox{ for } i = 1, \ldots, n$.

- A key thing in regression modeling is to describe the typical range of values of the outcome variable, given the predictors

- This is done in two steps:
    1. Predict the average value of the outcome given the predictors
    2. Summarize the variation in this prediction

- Probabilistic distributions are used in regression modeling to help us characterize the variation that remains ___after___ predicting the average

- This helps us get an understanding of how uncertain our predictions are

- And also how uncertain we are in our estimated parameters of the model

#### __The Mean and Standard Deviation of a Probability Distribution__

- A probability distribution of a random variable $z$ takes on some range of values (the numbers written on the balls drawn from the urn)

- The ___mean___ (or _expected value_) of this distribution is the average of all these numbers or the number that would be obtained on average from a random sample from the distribution

- $E(z) = \mu_{z}$ is how we write the expected value in symbols

- See figure 3.6 for a distribution of women's heights in the United States

- The mean is $63.7$ inches:
    - This is the average height of all women in the country
    - It is also the value we would expect to observe by sampling one woman at random from the population

- The ___variance___ of the distribution of $z$ is $E[(z - \mu_{z})^{2}]$

- The mean of the squared difference from the mean

- The ___standard deviation___ is the square root of the variance

- We typically work with the standard deviation rather than variance because it is in terms of the scale of the random variable $z$

- From Figure 3.6 the standard deviation of women's heights in the United States is 2.7 inches

- The variance of 7.3 is on the uninterpretable scale of inches squared

## __Section 3.6: Probability Modeling__

```python

```

```python

```

```python

```

## __Section 3.7: Bibliographic Note__

```python

```