# Introduction to Fourier Series and Transforms

**Student Version**

### This notebook contains both class and exercice materials. Classes material will be written in black, examples in blue, and problems to solve in red.

* * *

By L. Bugnet (Flatiron Institute)

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from ipywidgets import interact, fixed
from scipy.fftpack import fft, ifft, rfft
import time
from scipy import signal
from lightkurve import search_targetpixelfile
import lightkurve as lk
import astropy.units as u

%matplotlib notebook
plt.ion()
```

    <matplotlib.pyplot._IonContext at 0x1c192f62490>

# Part 1: Fourier Series and signal decomposition

```python
from IPython.display import Audio,Image, YouTubeVideo
id='r18Gi8lSkfM'
#YouTubeVideo(id=id,width=1000,height=600)
```

## Total Fourier Series formula:

Periodic signal f(x) with period $T=2\pi/\omega$

\begin{equation}
\boxed{F(x) = A_0 + \sum_{n=1}^\infty A_n \cos(n\omega x -\Phi_n)\,}
\end{equation}

That can be rewritten as:
\begin{equation}
\boxed{F(x) = a_0 + \sum_{n=1}^\infty a_n \cos(n\omega x) + \sum_{n=1}^\infty b_n \sin(n\omega x)\,}
\end{equation}

with
\begin{equation}
a_0 = \frac{1}{T} \int_0^T f(x)
\end{equation}
\begin{equation}
a_n = \frac{2}{T} \int_0^T f(x)\cos(n \omega x)
\end{equation}
\begin{equation}
b_n = \frac{2}{T} \int_0^T f(x) \sin(n \omega x)
\end{equation}

### Example 1.1: Approximate f with a Fourier Series
\begin{equation}
f(x) =
    \begin{cases}
      1 & \text{0<x<$\pi$ $[2\pi]$}\\
      0 & \text{$\pi$<x<2$\pi$ $[2\pi]$}
    \end{cases}
\end{equation}
f is periodic of Period $T=2\pi$, $\omega=1$ so:

\begin{equation}
    a_0=\frac{1}{T}\int_0^\pi 1 {\rm d}x \left(+\frac{1}{T}\int_\pi^{2\pi} 0 {\rm d}x\right) = \frac{1}{2}
    \end{equation}

\begin{equation}
    a_n=\frac{1}{T}\int_0^\pi \cos{n x} {\rm d}x = \frac{\sin{n\pi}}{n\pi}
    \end{equation}

\begin{equation}
    b_n=\frac{1}{T}\int_0^\pi \sin{n x} {\rm d}x = \frac{1-\cos{n\pi}}{n\pi}
    \end{equation}

Let's try to approximate $f$ by:

\begin{equation}
    F(x)=\frac{1}{2}+\sum_{n=1}^\infty\left(\frac{\sin{n\pi}}{n\pi}\cos(nx) +\frac{\sin{n\pi}}{n\pi}\sin(nx)\right)
    \end{equation}

```python
#Plot f over 3T-wide intervals:
x1=np.linspace(0, np.pi, 100)
x2=np.linspace(np.pi, 2*np.pi, 100)

plt.figure()
for ii in range(0,4):
    plt.plot(x1+2*np.pi*ii, np.ones(len(x1)), color='k', lw=5)
    plt.plot(x2+2*np.pi*ii, np.zeros(len(x2)), color='k', lw=5)

#Overplot Fourier Serie F:
x = np.linspace(0, 8*np.pi, 800)
a0=1./2.
Na = [ 1, 3, 5, 11, 101] #number of sine function to consider in the Fourier Serie

#Fucntion that calculates F from a0, an, bn:
def f_N(x,N, a0):
    na = range(1,N+1,2)
    f_N = np.zeros(x.shape)+a0
    for n in na:
        an = np.sin(n*np.pi)/(n*np.pi)
        bn = (1-np.cos(n*np.pi))/(n*np.pi)
        f_N += an*np.cos(n*x) + bn*np.sin(n*x)
    return f_N

for N in Na:
    plt.plot(x,f_N(x,N, a0),label='N={}'.format(N), alpha=0.5)
plt.xlabel('x')
plt.legend(loc='best')

```

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x1c19a211ee0>

### Example 1.2: Approximate continuous function with a Fourier Series
\begin{equation}
f(x) =x(1-x)\,\text{with } 0<x<1
\end{equation}

```python
x = np.linspace(0,1,1000)

# small python function to define the partial sum for the truncated Fourier Series f_N
def f_N(x,N):
    """
    small python function to define the partial sum for the truncated Fourier Series f_N
    """
    na = range(1,N+1,2)
    f_N = np.zeros(x.shape)
    for n in na:
        f_N += 8/((n*np.pi)**3)*np.sin(n*np.pi*x)
    return f_N

# And make a figure showing f_N for increasing values of N
Na = [ 1, 3, 5, 11, 101]

plt.figure()
for N in Na:
    plt.plot(x,f_N(x,N),label='N={}'.format(N))
plt.plot(x,x*(1-x),'k',label='$f(x)=x(1-x)$', lw=5)
plt.xlabel('x')
plt.legend(loc='best')
plt.grid()
plt.show()
```

    <IPython.core.display.Javascript object>

### Example 1.3: Approximate 2D function with a Fourier Series

```python
def fourier2D_xy(m_terms=5, n_terms=5):
    """Plot the 2D Fourier approximation for a hyperbolic paraboloid

    m_terms, and n_terms are the number of terms in x and y.

    The values are padded to be between [-0.9 pi, 0.9 pi] to avoid the
    discontinuities in the border of the domain.
    """
    Y, X = 0.9*np.pi * np.mgrid[-1:1:21j, -1:1:21j]
    XY = np.zeros_like(X)
    for cont_x in range(1, m_terms + 1):
        for cont_y in range(1, n_terms + 1):
            XY = XY + (-1)**(cont_x + cont_y) * \
                np.sin(cont_x*X) * np.sin(cont_y*Y)/(cont_x*cont_y)

    XY = 4*XY

    fig = plt.figure(figsize=(6, 4))
    ax = fig.add_subplot(111, projection='3d')
    ax.plot_surface(X, Y, XY, cmap="Purples_r", cstride=1, rstride=1, alpha=0.8,
                   lw=0.2)
    ax.plot_wireframe(X, Y, X*Y, cstride=1, rstride=1, color="#e41a1c",
                     linewidth=1)

interact(fourier2D_xy,
     m_terms=(1, 500),
     n_terms=(1, 500));

interact(fourier2D_xy(m_terms=500, n_terms=500),
     m_terms=(1, 500),
     n_terms=(1, 500));
```

    interactive(children=(IntSlider(value=5, description='m_terms', max=500, min=1), IntSlider(value=5, descriptio…

    <IPython.core.display.Javascript object>

Visit link for visual interpretation of signal from sin/cos: https://www.geogebra.org/m/ztn2du2a

### What if the signal f is not periodic?

* #### Fourier series

Periodic function => converts into a discrete exponential or sine and cosine function.

Non-periodic function => not applicable
* #### Fourier transform

Periodic function => converts its Fourier series in the frequency domain.

Non-Periodic function => converts it into continuous frequency domain.

```python
from IPython.display import Audio,Image, YouTubeVideo
id='r18Gi8lSkfM'
#YouTubeVideo(id=id,width=1000,height=600)
# start at 9:11
```

# Part 2: Fourier Transforms

"The Fourier transform is a way of splitting something up into a bunch of sine waves"

#### Fourier transform of a function f definition:
\begin{equation}
\boxed{F(\nu) = \int_{-\infty}^{\infty} f(x)e^{-2\pi i\nu x} {\rm d}x}
\end{equation}

### Example 2.1: Get the fourier transform of f
 \begin{equation}
f(x) = \cos(2\pi 3x)
\end{equation}

```python
#plot function f:
x = np.linspace(-3,3, 1000)
plt.figure()
plt.plot(x,np.cos(2*np.pi*x*3), alpha=1 , color='dodgerblue')
plt.xlabel('x')
plt.ylabel('f(x)')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'f(x)')

f is periodic of known period $T=\frac{1}{3}$. Let's calculate recover this periodicity by using the Fourier transform of f:

\begin{equation}
    F(\nu) = \int_{-\infty}^{\infty}\cos(2\pi 3x)e^{-2\pi i \nu x}{\rm d}x
    \end{equation}

\begin{equation}
    F(\nu) = \int_{-\infty}^{\infty}\left(\frac{e^{2\pi i 3 x}+e^{-2 \pi i 3 x}}{2}\right)e^{-2\pi i \nu x}{\rm d}x
    \end{equation}

\begin{equation}
    F(\nu) = \int_{-\infty}^{\infty}\left(\frac{e^{2\pi i (3 x-\nu x)}+e^{-2 \pi i (3 x + \nu x)}}{2}\right){\rm d}x
    \end{equation}

\begin{equation}
\boxed{F(\nu) = \frac{1}{2}\left(\delta\left(\nu-3)\right) +\delta\left(\nu+3\right)\right)}
\end{equation}

The Fourier transform of $f(x) = \cos(2\pi 3x)$ is zero except at the frequency $\nu=3$ (and $\nu =-3$ not acceptable for most physical problems), corresponding to a periodicity of $T=\frac{1}{\nu}=\frac{1}{3}$

### As shown by this example, if the function is periodic the Fourier transform is a sum of Dirac positionned at the different frequencies.

### Problem 2.2: Get the fourier transform of f
 \begin{equation}
f(x) = \sin(2\pi 3x)
\end{equation}

Enter details of calculation here!

Solution:

\begin{equation}
F(\nu) = \frac{1}{2i}\left(\delta\left(\nu-3)\right) +\delta\left(\nu+3\right)\right)
\end{equation}

### Conclusion:
* ### Even functions have Real Fourier Series
* ### Odd functions have Imaginary Fourier Series

* ### In general, we use the absolute value of the FT to take into account both odd and even components in the function: |FT|

### What if the function is not periodic?

### Problem 2.3: Fourier transform of Periodic VS non-periodic function

Represent the following functions:

\begin{equation}
f_1(x) = \sin(6\pi x)
\end{equation}

\begin{equation}
f_2(x) = \sin(6\pi x)e^{-2\pi x}
\end{equation}

```python
#plot function f:
x = np.linspace(-3,3, 1000)
plt.figure()
plt.plot(x,np.cos(2*np.pi*x*3), alpha=1 , color='dodgerblue')
plt.xlabel('x')
plt.ylabel('f(x)')
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'f(x)')

f is periodic of known period $T=\frac{1}{3}$. Let's calculate recover this periodicity by using the Fourier transform of f:

\begin{equation}
    F(\nu) = \int_{-\infty}^{\infty}\cos(2\pi 3x)e^{-2\pi i \nu x}{\rm d}x
    \end{equation}

\begin{equation}
    F(\nu) = \int_{-\infty}^{\infty}\left(\frac{e^{2\pi i 3 x}+e^{-2 \pi i 3 x}}{2}\right)e^{-2\pi i \nu x}{\rm d}x
    \end{equation}

\begin{equation}
    F(\nu) = \int_{-\infty}^{\infty}\left(\frac{e^{2\pi i (3 x-\nu x)}+e^{-2 \pi i (3 x + \nu x)}}{2}\right){\rm d}x
    \end{equation}

\begin{equation}
\boxed{F(\nu) = \frac{1}{2}\left(\delta\left(\nu-3)\right) +\delta\left(\nu+3\right)\right)}
\end{equation}

The Fourier transform of $f(x) = \cos(2\pi 3x)$ is zero except at the frequency $\nu=3$ (and $\nu =-3$ not acceptable for most physical problems), corresponding to a periodicity of $T=\frac{1}{\nu}=\frac{1}{3}$

### As shown by this example, if the function is periodic the Fourier transform is a sum of Dirac positionned at the different frequencies.

### Problem 2.2: Get the fourier transform of f
 \begin{equation}
f(x) = \sin(2\pi 3x)
\end{equation}

Enter details of calculation here!

Solution:

\begin{equation}
F(\nu) = \frac{1}{2i}\left(\delta\left(\nu-3)\right) -\delta\left(\nu+3\right)\right)
\end{equation}

### Conclusion:
* ### Even functions have Real Fourier Series
* ### Odd functions have Imaginary Fourier Series

* ### In general, we use the absolute value of the FT to take into account both odd and even components in the function: |FT|

### What if the function is not periodic?

### Problem 2.3: Fourier transform of Periodic VS non-periodic function

Represent the following functions:

\begin{equation}
f_1(x) = \sin(6\pi x)
\end{equation}

\begin{equation}
f_2(x) = \sin(6\pi x)e^{-2\pi x}
\end{equation}

```python
x = np.linspace(0,3, 10000)
fig, ax = plt.subplots(1,2, figsize=(9,4))

#plot function f1:
ax[0].plot(x, np.sin(6*np.pi*x), alpha=1 , color='crimson')

ax[0].set_xlabel('x')
ax[0].set_ylabel(r'$f_1(x)$')

#plot function f2:
ax[1].plot(x, np.sin(6*np.pi*x)*np.exp(-2*np.pi*x), alpha=1 , color='crimson')

ax[1].set_xlabel('x')
ax[1].set_ylabel(r'$f_2(x)$')

plt.tight_layout();

```

    <IPython.core.display.Javascript object>

and represent their fourier transforms:

\begin{equation}
F_1(\nu) = \frac{1}{2i}\left(\delta\left(\nu-3)\right) +\delta\left(\nu+3\right)\right)
\end{equation}

\begin{equation}
F_2(\nu) = \frac{6 \pi}{(2\pi+2\pi\nu i)^2+(6\pi)^2}
\end{equation}

```python
nu = np.linspace(0,50, 10000)

#we approximate FT1 with:
FT1 = np.zeros(10000)
FT1[np.where(np.abs(nu-(3))==np.min(np.abs(nu-3)))]=1

#plot FT1:
fig, ax = plt.subplots(1,2, figsize=(9,4))
ax[0].plot(nu, np.abs(FT1), color='crimson')
wh1 = np.argmax(np.abs(FT1))
ax[0].plot(nu[wh1], np.abs(FT1)[wh1], '+', color='crimson', markersize=10)
ax[0].set_xlabel(r'$\nu$')
ax[0].set_ylabel(r'F($\nu$)')

#plot FT2
FT2 = 6*np.pi/((2*np.pi*(1+nu*1j))**2 + (6*np.pi)**2)
ax[1].plot(nu, np.abs(FT2), color='crimson')
ax[1].set_xlabel(r'$\nu$')
ax[1].set_ylabel(r'F($\nu$)')

#plot maximum of FT2, extract Period of f2
wh2 = np.argmax(np.abs(FT2))
ax[1].plot(nu[wh2], np.abs(FT2)[wh2], '+', color='crimson', markersize=10)

print('T2='+str(1/nu[wh2]))
```

    <IPython.core.display.Javascript object>

    T2=0.353321554770318

### Conclusion: If the function is not periodic in the time domain, the FT converts it to a continuous function in the frequency domain:

### What if the signal is discrete?

# Part 3: Discrete Fourier Transforms (DFT)

Discrete Fourier tranform are used on discrete signals.

\begin{equation}
\boxed{X_k = \sum_{n=0}^{N-1}x_n \displaystyle  e^{\displaystyle -2 i \pi  \frac{kn}{N}}}
\end{equation}

with
* N = number of samples
* n = current sample
* k = current frequency, where k∈[0,𝑁−1]
* $x_n$ = the sine value at sample n
* $X_k$ = The DFT which include information of both amplitude and phase:

\begin{equation}
A = \frac{|X_k|}{N}
\end{equation}

\begin{equation}
\Phi = \arctan{\frac{\mathcal{Im}(X_k)}{\mathcal{Re}(X_k)}}
\end{equation}

Visit link for visual interpretation of Discrete Fourier transform: https://www.geogebra.org/m/ztn2du2a

### Problem 3.1: Write a function DFT(x) which takes in one argument, x - input 1 dimensional real-valued signal. The function will calculate the DFT of the signal and return the DFT values.

```python
### Solution:
def DFT(x):
    """
    Function to calculate the
    discrete Fourier Transform
    of a 1D real-valued signal x
    """
    N = len(x)
    X = np.full(N,0j)
    for k in range(N):
        for n in range(N):
            X[k] += x[n]*np.exp(-2*np.pi*1j*n*k/N)

    return X

print(DFT(np.sin(np.arange(3)*2*np.pi/3 + 0.5)))
```

    [1.11022302e-16+0.j         7.19138308e-01-1.31637384j
     7.19138308e-01+1.31637384j]

Apply this DFT function to the discrete signal generated by $f(x) = \sin(6\pi x)e^{-2\pi x}$ (see above) with a sampling rate of 1000 (x=np.linspace(0,10,1000)) and plot the result.

```python
sr =1000 #sampling rate
x = np.linspace(0,10, sr)
y = np.sin(6*np.pi*x)*np.exp(-2*np.pi* x)
X = DFT(y)
# calculate the frequencies associated with x:
freq = np.linspace(0,1/(10/sr),sr)

fig, ax=plt.subplots(1,2, figsize=(10,4))
ax[0].plot(x, y, 'o',color='crimson', markersize=3)
ax[0].set_xlabel('x')
ax[0].set_ylabel('y')

ax[1].plot(freq, abs(X), 'o',color='crimson', markersize=3)

ax[1].set_xlabel(r'$\nu$')
ax[1].set_ylabel(r'|DFT($\nu$)|')

#calculate Periodicity
whd = np.argmax(np.abs(X))
print('T='+str(1/freq[whd]))
```

    <IPython.core.display.Javascript object>

    T=0.35678571428571426

### Do you recover $T\approx 0.35$?

### Why is the DFT mirrored?

    see handwritten notes, periodicity and mirroring (due to real signal) mean N/2 unique values in X

# Part 4: Fast Fourier Transform

####  Fast Fourier Transforms are a family of algorithm that computes the DFT efficeintly. They are used to reduce the computational time from $\mathcal{O}(N^2)$ to $\mathcal{O}(N\log{N})$
 See https://www.ece.ucdavis.edu/~bbaas/281/papers/CooleyLewisWelch.1967.HistNotesFFT.pdf for details about the different FFT algorithms.

Of course, many python packages like scipy and numpy already contain implemented FFT functions, so that you don't have to.

Let's compare the FFT with our DFT function:

```python
#Calculate the FFT of our data
x = np.linspace(0,10, sr)
FFT = np.fft.fft(np.sin(6*np.pi*x)*np.exp(-2*np.pi*x))

#Calculate the associated frequencies
freqs = np.fft.fftfreq(len(x), d=x[1]-x[0])

#Plot FFT
plt.figure()
plt.plot(freqs,np.abs(FFT), 'o', color='dodgerblue', markersize=3)
plt.xlabel(r'$\nu$')
plt.ylabel(r'FFT($\nu$)')
plt.xlim(0,50)
wh = np.argmax(np.abs(FFT))
plt.plot(freqs[wh], np.abs(FFT)[wh], '+', color='crimson', markersize=10)

#Extract dominant frequency
print('T='+str(1/freqs[wh]))

```

    <IPython.core.display.Javascript object>

    T=0.35750035750035747

### Is FFT really more efficient than our home made DFT function ?

## Problem 4.1: Computational time: DFT VS FFT

Compare the computational time of the DFT function and the FFT algorithm on the discrete function $f(x) = \sin(6\pi x)e^{-2\pi x}$ with two different observation cadence:

x = np.linspace(0,10, 1000)

x = np.linspace(0,10, 10000)

```python
#sampling rate:
sr = 1000

# computation time with our DFT function:
x=np.linspace(0,10,1000)
y=np.sin(6*np.pi*x)*np.exp(-2*np.pi*x)
t1=time.time()
DFT(y)
t2=time.time()
print('DFT 1000 samples computational time='+ str(t2-t1) +' sec')

#increase number of point (*10):
x=np.linspace(0,10,10000)
y=np.sin(6*np.pi*x)*np.exp(-2*np.pi*x)
t1=time.time()
#DFT(y)
t2=time.time()
#print('DFT 10000 samples computational time='+ str(t2-t1) +' sec')

# computation time with FFT
x=np.linspace(0,10,1000)
y=np.sin(6*np.pi*x)*np.exp(-2*np.pi*x)
t1=time.time()
np.fft.fft(y)
t2=time.time()
print('FFT 1000 samples computational time='+ str(t2-t1) +' sec')

#increase number of point (*10):
x=np.linspace(0,10,10000)
y=np.sin(6*np.pi*x)*np.exp(-2*np.pi*x)
t1=time.time()
np.fft.fft(y)
t2=time.time()
print('FFT 10000 samples computational time='+ str(t2-t1) +' sec')

#results:
#DFT 1000 samples computational time=3.657241106033325 sec
#DFT 10000 samples computational time=361.37832403182983 sec
#FFT 1000 samples computational time=0.0 sec
#FFT 10000 samples computational time=0.0 sec

```

    DFT 1000 samples computational time=14.098427534103394 sec
    FFT 1000 samples computational time=0.000982046127319336 sec
    FFT 10000 samples computational time=0.0009520053863525391 sec

### Conclusion: Use FFT algorithms!

# Part 5: Reverse fourier transform:

\begin{equation}
\boxed{f(x)=\int_{-\infty}^{\infty} F(\nu)e^{\displaystyle 2\pi i \nu x}{\rm d}\nu}
\end{equation}

#### Discrete version:

\begin{equation}
\boxed{x_n=\frac{1}{N}\sum_{k=0}^{N-1} X_k e^{\displaystyle i\frac{2\pi}{N}kn}}
\end{equation}

```python
# Reverse the discrete Fourier transform:
ifft_2 = ifft(FFT)
x=np.linspace(0,10,1000)
fig, ax = plt.subplots(1, 2, figsize=(9,4))
ax[0].plot(x, np.sin(6*np.pi*x)*np.exp(-2*np.pi* x), label='f(x)', color='dodgerblue')
ax[0].plot(x, ifft_2, '--', label='iFT(FT(f(x)))', color='crimson')
ax[0].set_xlabel('x')
ax[0].legend()

ax[1].plot(x, np.sin(6*np.pi*x)*np.exp(-2*np.pi* x)-ifft_2, color='dodgerblue')
ax[1].set_xlabel('x')
ax[1].set_ylabel('f(x)-iFT(FT(x))')
plt.tight_layout()

```

    <IPython.core.display.Javascript object>

    C:\Users\matth\miniconda3\envs\DSFP\lib\site-packages\numpy\core\_asarray.py:102: ComplexWarning: Casting complex values to real discards the imaginary part
      return array(a, dtype, copy=False, order=order)
    C:\Users\matth\miniconda3\envs\DSFP\lib\site-packages\numpy\core\_asarray.py:102: ComplexWarning: Casting complex values to real discards the imaginary part
      return array(a, dtype, copy=False, order=order)

# Part 6: Observation effects

### Example 6.1: Calculate the FFT of the sin function defined on the following intervals:

```python
fig, ax =plt.subplots(1,3, figsize=(10,4))
fig2, ax2 =plt.subplots(1,3, figsize=(10,4))

#1st interval
x1 = np.linspace(-2*np.pi,3.5*np.pi,1000)

y1 = np.sin(x1)
ax[0].plot(x1,y1, color='dodgerblue')
FFT1 = np.fft.fft(y1)
nu1 = np.fft.fftfreq(len(x1), d=x1[1]-x1[0])
ax2[0].plot(nu1, np.abs(FFT1), 'o', color='dodgerblue', markersize=5)
wh1 = np.argmax(np.abs(FFT1))
ax2[0].plot(nu1[wh1], np.abs(FFT1)[wh1], '+', color='crimson', markersize=10)
print('Period from FFT 1 = '+str(1/nu1[wh1]))

#2nd interval
x2 = np.linspace(-2*np.pi,35*np.pi,1000)

y2 = np.sin(x2)
ax[1].plot(x2,y2, color='dodgerblue')
FFT2 = np.fft.fft(y2)
nu2 = np.fft.fftfreq(len(x2), d=x2[1]-x2[0])
ax2[1].plot(nu2, np.abs(FFT2), 'o', color='dodgerblue', markersize=5)
ax2[1].set_xlim(0,1)
wh2 = np.argmax(np.abs(FFT2))
ax2[1].plot(nu2[wh2], np.abs(FFT2)[wh2], '+', color='crimson', markersize=10)
print('Period from FFT 2 = '+str(1/nu2[wh2]))

#3rd interval
x3 = np.linspace(-2*np.pi,64*np.pi,1000)

y3 = np.sin(x3)
ax[2].plot(x3,y3, color='dodgerblue')
FFT3 = np.fft.fft(y3)
nu3 = np.fft.fftfreq(len(x3), d=x3[1]-x3[0])
ax2[2].plot(nu3, np.abs(FFT3), 'o', color='dodgerblue', markersize=5)
wh3 = np.argmax(np.abs(FFT3))
ax2[2].plot(nu3[wh3], np.abs(FFT3)[wh3], '+', color='crimson', markersize=10)
print('Period from FFT 3 = '+str(1/nu3[wh3]))
print('True Period= '+str(2*np.pi))

ax2[0].set_xlim(0,1)
ax2[1].set_xlim(0,1)
ax2[2].set_xlim(0,1)

ax[0].set_xlabel('x')
ax[0].set_ylabel('sin(x)')
ax[1].set_xlabel('x')
ax[2].set_xlabel('x')

ax2[0].set_xlabel(r'$\nu$')
ax2[0].set_ylabel('FFT 1(sin(x))')
ax2[1].set_xlabel(r'$\nu$')
ax2[1].set_ylabel('FFT 2(sin(x))')
ax2[2].set_xlabel(r'$\nu$')
ax2[2].set_ylabel('FFT 3(sin(x))')

fig.tight_layout()
fig2.tight_layout()

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    Period from FFT 1 = 5.765351883464832
    Period from FFT 2 = -6.123962287699379
    Period from FFT 3 = 6.289474781961553
    True Period= 6.283185307179586

### Conclusion:
* The duration of the time serie matters: the longest the time serie, the better the frequency resolution in the FFT (see panel 2 VS panel 3).
* When the number of periods in the acquisition is not an integer, the endpoints are discontinuous. These artificial discontinuities show up in the FFT as high-frequency components not present in the original signal (see panel 1 VS panel 2).

## PSD and amplitudes

Check out how the longer the time series is (and the finer the frequency bin width/step is), the higher the amplitude of the FFT becomes! This means you can't really use FFTs to compare and quantify vibration environments.

Power spectrum densities (PSDs) are normalized to the frequency bin width preventing the duration of the data set (and corresponding frequency step) from changing the amplitude of the result. The definition the power spectral density of a continuous function is the Fourier transform of its auto-correlation function

\begin{equation}
{\rm PSD}(\nu)=\lim_{T\rightarrow \infty}\frac{1}{T}|{\rm FT}(\nu)|^2
\end{equation}

```python
rng = np.random.default_rng()
fs = 10e3
N = 1e3
amp = 2*np.sqrt(2)
freq = 1234.0
noise_power = 0.001 * fs / 2
time = np.arange(N) / fs
x = amp*np.sin(2*np.pi*freq*time)
f, Pxx_den = signal.welch(x, fs, nperseg=1024)
fig, ax = plt.subplots(2,3, figsize=(9,6))
ax[0,1].semilogy(f, Pxx_den, color='dodgerblue')
ax[0,1].set_ylim([0.5e-3, 1])
ax[0,1].set_xlabel('frequency [Hz]')
ax[0,1].set_ylabel('PSD')
ax[0,2].set_ylabel('FFT')
ax[0,2].plot(np.fft.fftfreq(len(x), d=time[1]-time[0]), np.abs(np.fft.fft(x)), color='dodgerblue')
ax[0,2].set_xlim(0, np.max(np.fft.fftfreq(len(x), d=time[1]-time[0])))
ax[0,0].plot(x, color='dodgerblue', lw=0.1)

N = 1e5
time = np.arange(N) / fs
x = amp*np.sin(2*np.pi*freq*time)
f, Pxx_den = signal.welch(x, fs, nperseg=1024)
ax[1,1].semilogy(f, Pxx_den, color='dodgerblue')
ax[1,1].set_ylim([0.5e-3, 1])
ax[1,1].set_xlabel('frequency [Hz]')
ax[1,1].set_ylabel('PSD')
ax[1,2].set_ylabel('FFT')
ax[1,2].plot(np.fft.fftfreq(len(x), d=time[1]-time[0]), np.abs(np.fft.fft(x)), color='dodgerblue')
ax[1,2].set_yscale('log')
ax[1,2].set_xlim(0, np.max(np.fft.fftfreq(len(x), d=time[1]-time[0])))
ax[1,0].plot(x, color='dodgerblue', lw=0.01)
plt.tight_layout()
```

    C:\Users\matth\miniconda3\envs\DSFP\lib\site-packages\scipy\signal\spectral.py:1961: UserWarning: nperseg = 1024 is greater than input length  = 1000, using nperseg = 1000
      warnings.warn('nperseg = {0:d} is greater than input length '

    <IPython.core.display.Javascript object>

### Problem 6.2: Effect of gaps in data

 Compare the FFT of the following data set and the FFT obtained when removing some data points.
 What is the effect of gaps in the FFT?

```python
fig, ax =plt.subplots(1,2, figsize=(10,4))
fig2, ax2 =plt.subplots(1,2, figsize=(10,4))

#1st interval
x = np.linspace(-2*np.pi,64*np.pi,1000)
y = np.sin(x)
ax[0].plot(x,y, color='crimson')
FFT = np.fft.fft(y)
nu = np.fft.fftfreq(len(x),x[1]-x[0])
ax2[0].plot(nu, np.abs(FFT), 'o', color='crimson', markersize=5)

#calculate period
period1 = 1/nu[np.argmax(FFT[0:len(FFT)//2])]
print('Period from FFT 1 = '+str(period1))

#Create artificial gaps in the data
y[np.where((x>0) & (x<8*np.pi))]=0
y[np.where((x>15*np.pi) & (x<20*np.pi))]=0
y[np.where((x>45*np.pi) & (x<46*np.pi))]=0

ax[1].plot(x,y, color='crimson')
FFT = np.fft.fft(y)
nu = np.fft.fftfreq(len(x),x[1]-x[0])
ax2[1].plot(nu, np.abs(FFT), 'o', color='crimson', markersize=5)
ax2[1].plot(nu, np.abs(FFT),  color='crimson', markersize=5)

#calculate period
period2 = 1/nu[np.argmax(FFT[0:len(FFT)//2])]
print('Period from FFT 2 = '+str(period2))

ax2[0].set_xlim(0,1)
ax2[1].set_xlim(0,1)

ax[0].set_xlabel('x')
ax[0].set_ylabel('y')
ax[1].set_xlabel('x')

ax2[0].set_xlabel(r'$\nu$')
ax2[0].set_ylabel('FFT(y)')
ax2[1].set_xlabel(r'$\nu$')

fig.tight_layout()
fig2.tight_layout()
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    Period from FFT 1 = 6.289474781961553
    Period from FFT 2 = 6.104490229550919

# Part 7: Let's play with real data!

### We are gonna try to extract the rotation period of a solar-type star observed by Kepler.

Its identifier is KIC 3541346, and has been studied by Vida+ 2014: https://arxiv.org/abs/1404.4359

Kepler observed a region in the sky during about 4 years, divided into quarters.

We are using the Lightkurve suite (Lightkurve Collaboration 2018). Documentation: https://docs.lightkurve.org

```python
# ---excecute this cell---

#Download KIC 3541346 lightcurve (flux VS time):
search_result = lk.search_lightcurve('KIC 3541346', author='Kepler', cadence='long')
print(search_result)

```

    SearchResult containing 17 data products.

     #       mission      year author exptime  target_name  distance
                                         s                   arcsec
    --- ----------------- ---- ------ ------- ------------- --------
      0 Kepler Quarter 01 2009 Kepler    1800 kplr003541346      0.0
      1 Kepler Quarter 02 2009 Kepler    1800 kplr003541346      0.0
      2 Kepler Quarter 03 2009 Kepler    1800 kplr003541346      0.0
      3 Kepler Quarter 04 2010 Kepler    1800 kplr003541346      0.0
      4 Kepler Quarter 05 2010 Kepler    1800 kplr003541346      0.0
      5 Kepler Quarter 06 2010 Kepler    1800 kplr003541346      0.0
      6 Kepler Quarter 07 2010 Kepler    1800 kplr003541346      0.0
      7 Kepler Quarter 08 2011 Kepler    1800 kplr003541346      0.0
      8 Kepler Quarter 09 2011 Kepler    1800 kplr003541346      0.0
      9 Kepler Quarter 10 2011 Kepler    1800 kplr003541346      0.0
     10 Kepler Quarter 11 2012 Kepler    1800 kplr003541346      0.0
     11 Kepler Quarter 12 2012 Kepler    1800 kplr003541346      0.0
     12 Kepler Quarter 13 2012 Kepler    1800 kplr003541346      0.0
     13 Kepler Quarter 14 2012 Kepler    1800 kplr003541346      0.0
     14 Kepler Quarter 15 2013 Kepler    1800 kplr003541346      0.0
     15 Kepler Quarter 16 2013 Kepler    1800 kplr003541346      0.0
     16 Kepler Quarter 17 2013 Kepler    1800 kplr003541346      0.0

The star has been observed during 17 quarters. Let's represent its lightkurve.

```python
# ---Excecute this cell---
#light curve representation:
lc_collection = search_result.download_all()
#we stitch all the quarters together:
lc_stitched = lc_collection.stitch()
lc_stitched.plot();

```

    <IPython.core.display.Javascript object>

#### Can we apply a FFT directly on this data?

---> No because time array is not evenly spaced.

---> Fill time gaps with 1.00 values to create regular time array

```python
# ---Fill and excecute this cell---

#extract time and flux from lightcurve:
time = lc_stitched.time.to_value(format='jd')
flux = lc_stitched.flux.to_value()

# create new evenly-spaced time/flux arrays (time_even, flux_even):
# Tip: plot np.diff(time):
plt.figure()
plt.plot(np.diff(time), color='dodgerblue')

#chose threshold:
wh = np.where(np.diff(time)>0.03)

dt=time[1]-time[0]

time_even=[]
flux_even=[]

for ii in range(len(time[1:])):
    if time[ii]-time[ii-1]<0.03:
        time_even=np.append(time_even, [time[ii]], axis=0)
        flux_even=np.append(flux_even, [flux[ii]], axis=0)

    else:
        ind = 0
        while time[ii]-(time[ii-1]+ind* dt)>0.03:
            ind+=1
            time_even=np.append(time_even, [time[ii-1]+ind* dt], axis=0)
            flux_even=np.append(flux_even, [1], axis=0)
        if time[ii]-(time[ii-1]+ind* dt)<=0.03:
            time_even=np.append(time_even, [time[ii]], axis=0)
            flux_even=np.append(flux_even, [flux[ii]], axis=0)

#check that new time_even is ~ regularly spaced
plt.plot(np.diff(time_even), color='orange')
plt.ylabel('Time diff [days]')

#we artificially make the time array perfectly regular for this project. See Lomb-Scargle course for dealing with non-regular data.
time =time_even#[0:1000]
time_res=(time[-1]-time[0])/len(time)
time = [time[0]+ii*time_res for ii in range(len(time))]
flux =flux_even#[0:1000]
flux = np.nan_to_num(flux, copy=True, nan=1, posinf=None, neginf=None)

#plot new lightcurve
plt.figure()
plt.plot(time, flux, color='dodgerblue')
plt.ylabel('Time [days]')
plt.ylabel('Flux')

```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'Flux')

### Then compute FFT of the time serie:

```python
# ---Fill and excecute this cell---
time_res=(time[1]-time[0])#/len(time)
# Number of sample points
N = len(flux)
# sample spacing
T = time_res

yf = np.fft.fft(flux)
xf = np.fft.fftfreq(N, T)[1:N//2]

fig, ax = plt.subplots(1,2, figsize=(9,4))
ax[0].set_yscale('log')
ax[0].plot(xf, np.abs(yf[1:N//2]), color='dodgerblue')

wh = np.argwhere((np.abs(yf[10:N//2])==np.max(np.abs(yf[10:N//2]))) )
frequency_max = xf[9:][wh]
Period = 1/frequency_max[0][0]
ax[0].plot(frequency_max[0][0],np.abs(yf[10:N//2])[wh], '+', color='dodgerblue', markersize=5 )
ax[0].set_xlabel(r'Frequency [day$^{-1}$]')
ax[0].set_ylabel(r'FFT Amplitude')

print('Period from FFT1 = '+str(Period)+' days')

#compare with shorter time serie:
time2 = time[:130]
flux2 = flux[:130]
# Number of sample points
N = len(flux2)
# sample spacing
T = time_res

yf = np.fft.fft(flux2)
xf = np.fft.fftfreq(N, T)[1:N//2]

ax[1].set_yscale('log')
ax[1].plot(xf, np.abs(yf[1:N//2]), color='dodgerblue')

wh = np.argwhere((np.abs(yf[1:N//2])==np.max(np.abs(yf[1:N//2]))) )
frequency_max2 = xf[:][wh]
Period2 = 1/frequency_max2[0][0]
ax[1].plot(frequency_max2[0][0],np.abs(yf[1:N//2])[wh], '+', color='dodgerblue', markersize=5 )
ax[1].set_xlabel(r'Frequency [day$^{-1}$]')
ax[1].set_ylabel(r'FFT Amplitude')

print('Period from FFT2 = '+str(Period2)+' days')
```

    <IPython.core.display.Javascript object>

    Period from FFT1 = 0.908759920696634 days
    Period from FFT2 = 0.8854427250723044 days

### Let's check the FFT periodicity on the data:

```python
# ---excecute this cell, and check that your periodicity is correct:---

plt.figure()
plt.plot(time[:1300], flux[:1300], color='dodgerblue')
plt.vlines(np.array([time[0]+Period*ii for ii in range(30)]), 0.95, 1.1, linestyles ="dotted", colors ="k")
plt.xlabel('Time [days]')
plt.ylabel(r'Flux e$^-$/s')

```

    C:\Users\matth\AppData\Local\Temp/ipykernel_4260/240759100.py:3: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
      plt.figure()

    <IPython.core.display.Javascript object>

    Text(0, 0.5, 'Flux e$^-$/s')

#### What do other peaks represent?

## Your turn! Find the rotation period of KIC 10515986

```python
search_result = lk.search_lightcurve('KIC 10515986', author='Kepler', cadence='long')
print(search_result)

#light curve representation:
lc_collection = search_result.download_all()
#we stitch all the quarters together:
lc_stitched = lc_collection.stitch()
lc_stitched.plot();

```

    SearchResult containing 17 data products.

     #       mission      year author exptime  target_name  distance
                                         s                   arcsec
    --- ----------------- ---- ------ ------- ------------- --------
      0 Kepler Quarter 01 2009 Kepler    1800 kplr010515986      0.0
      1 Kepler Quarter 02 2009 Kepler    1800 kplr010515986      0.0
      2 Kepler Quarter 03 2009 Kepler    1800 kplr010515986      0.0
      3 Kepler Quarter 04 2010 Kepler    1800 kplr010515986      0.0
      4 Kepler Quarter 05 2010 Kepler    1800 kplr010515986      0.0
      5 Kepler Quarter 06 2010 Kepler    1800 kplr010515986      0.0
      6 Kepler Quarter 07 2010 Kepler    1800 kplr010515986      0.0
      7 Kepler Quarter 08 2011 Kepler    1800 kplr010515986      0.0
      8 Kepler Quarter 09 2011 Kepler    1800 kplr010515986      0.0
      9 Kepler Quarter 10 2011 Kepler    1800 kplr010515986      0.0
     10 Kepler Quarter 11 2012 Kepler    1800 kplr010515986      0.0
     11 Kepler Quarter 12 2012 Kepler    1800 kplr010515986      0.0
     12 Kepler Quarter 13 2012 Kepler    1800 kplr010515986      0.0
     13 Kepler Quarter 14 2012 Kepler    1800 kplr010515986      0.0
     14 Kepler Quarter 15 2013 Kepler    1800 kplr010515986      0.0
     15 Kepler Quarter 16 2013 Kepler    1800 kplr010515986      0.0
     16 Kepler Quarter 17 2013 Kepler    1800 kplr010515986      0.0

```python
time = lc_stitched.time.to_value(format='jd')
flux = lc_stitched.flux.to_value()

# create new evenly-spaced time/flux arrays (time_even, flux_even):
# Tip: plot np.diff(time):
plt.figure()
plt.plot(np.diff(time), color='dodgerblue')

#chose threshold:
wh = np.where(np.diff(time)>0.03)

dt=time[1]-time[0]

time_even=[]
flux_even=[]

for ii in range(len(time[1:])):
    if time[ii]-time[ii-1]<0.03:
        time_even=np.append(time_even, [time[ii]], axis=0)
        flux_even=np.append(flux_even, [flux[ii]], axis=0)

    else:
        ind = 0
        while time[ii]-(time[ii-1]+ind* dt)>0.03:
            ind+=1
            time_even=np.append(time_even, [time[ii-1]+ind* dt], axis=0)
            flux_even=np.append(flux_even, [1], axis=0)
        if time[ii]-(time[ii-1]+ind* dt)<=0.03:
            time_even=np.append(time_even, [time[ii]], axis=0)
            flux_even=np.append(flux_even, [flux[ii]], axis=0)

#check that new time_even is ~ regularly spaced
plt.plot(np.diff(time_even), color='orange')
plt.ylabel('Time diff [days]')

#we artificially make the time array perfectly regular for this project. See Lomb-Scargle course for dealing with non-regular data.
time =time_even#[0:1000]
time_res=(time[-1]-time[0])/len(time)
time = [time[0]+ii*time_res for ii in range(len(time))]
flux =flux_even#[0:1000]
flux = np.nan_to_num(flux, copy=True, nan=1, posinf=None, neginf=None)

#plot new lightcurve
plt.figure()
plt.plot(time, flux, color='dodgerblue')
plt.ylabel('Time [days]')
plt.ylabel('Flux')
```

```python
time_res=(time[1]-time[0])#/len(time)
# Number of sample points
N = len(flux)
# sample spacing
T = time_res

yf = np.fft.fft(flux)
xf = np.fft.fftfreq(N, T)[1:N//2]

fig, ax = plt.subplots(1,2, figsize=(9,4))
ax[0].set_yscale('log')
ax[0].plot(xf, np.abs(yf[1:N//2]), color='dodgerblue')

wh = np.argwhere((np.abs(yf[10:N//2])==np.max(np.abs(yf[10:N//2]))) )
frequency_max = xf[9:][wh]
Period = 1/frequency_max[0][0]
ax[0].plot(frequency_max[0][0],np.abs(yf[10:N//2])[wh], '+', color='dodgerblue', markersize=5 )
ax[0].set_xlabel(r'Frequency [day$^{-1}$]')
ax[0].set_ylabel(r'FFT Amplitude')

print('Period from FFT1 = '+str(Period)+' days')

#compare with shorter time serie:
time2 = time[:130]
flux2 = flux[:130]
# Number of sample points
N = len(flux2)
# sample spacing
T = time_res

yf = np.fft.fft(flux2)
xf = np.fft.fftfreq(N, T)[1:N//2]

ax[1].set_yscale('log')
ax[1].plot(xf, np.abs(yf[1:N//2]), color='dodgerblue')

wh = np.argwhere((np.abs(yf[1:N//2])==np.max(np.abs(yf[1:N//2]))) )
frequency_max2 = xf[:][wh]
Period2 = 1/frequency_max2[0][0]
ax[1].plot(frequency_max2[0][0],np.abs(yf[1:N//2])[wh], '+', color='dodgerblue', markersize=5 )
ax[1].set_xlabel(r'Frequency [day$^{-1}$]')
ax[1].set_ylabel(r'FFT Amplitude')

print('Period from FFT2 = '+str(Period2)+' days')
```

```python
plt.figure()
plt.plot(time[:1300], flux[:1300], color='dodgerblue')
plt.vlines(np.array([time[0]+Period*ii for ii in range(30)]), 0.95, 1.1, linestyles ="dotted", colors ="k")
plt.xlabel('Time [days]')
plt.ylabel(r'Flux e$^-$/s')
```

# Part 8: Challenge Problem! Determine the Mass and Radius of KIC 4682420

Credits IAC.

Ref: García & Ballot 2019

### 8.1) Represent the FFT of KIC 4682420 and identify the components.
### 8.2) Measure the frequency of maximum power. (numax)
### 8.3) Measure the large frequency separation. (deltanu)

### 8.4) Measure the Mass and Radius of the star by using the Asteroseismic scaling relations:

#### Asteroseismic scaling relations    (Bedding+ 2011):

\begin{equation}
\boxed{\frac{M}{{\rm M}_\odot}\approx\left(\frac{\nu_{\rm max}}{\nu_{{\rm max}_\odot}}\right)^3
\left(\frac{\Delta\nu}{\Delta\nu_\odot}\right)^{-4} \left(\frac{T_{\rm eff}}{T_{{\rm eff}_\odot}}\right)^{3/2} }
\end{equation}

\begin{equation}
\boxed{\frac{R}{{\rm R}_\odot}\approx\left(\frac{\nu_{\rm max}}{\nu_{{\rm max}_\odot}}\right)
\left(\frac{\Delta\nu}{\Delta\nu_\odot}\right)^{-2} \left(\frac{T_{\rm eff}}{T_{{\rm eff}_\odot}}\right)^{1/2} }
\end{equation}

with $$\nu_{{\rm max}_\odot} \approx 3090 {\rm μHz}$$ and $$\Delta\nu_{\odot} \approx 135.1 {\rm μHz}$$

```python

#Download KIC 4682420 lightcurve (flux VS time):
search_result = lk.search_lightcurve('KIC 4682420', author='Kepler', cadence='long')
print(search_result)

#represente light curve:
lc_collection = search_result.download_all()
#we stitch all the quarters together:
lc_stitched = lc_collection.stitch()
lc_stitched.plot();

#extract time and flux from lightcurve:
time = lc_stitched.time.to_value(format='jd')
flux = lc_stitched.flux.to_value()

# create new evenly-spaced time/flux arrays (time_even, flux_even):

plt.figure()
plt.plot(np.diff(time), color='dodgerblue')

#chose threshold:
wh = np.where(np.diff(time)>0.03)

dt=time[1]-time[0]

time_even=[]
flux_even=[]

for ii in range(len(time[1:])):
    if time[ii]-time[ii-1]<0.03:
        time_even=np.append(time_even, [time[ii]], axis=0)
        flux_even=np.append(flux_even, [flux[ii]], axis=0)

    else:
        ind = 0
        while time[ii]-(time[ii-1]+ind* dt)>0.03:
            ind+=1
            time_even=np.append(time_even, [time[ii-1]+ind* dt], axis=0)
            flux_even=np.append(flux_even, [1], axis=0)
        if time[ii]-(time[ii-1]+ind* dt)<=0.03:
            time_even=np.append(time_even, [time[ii]], axis=0)
            flux_even=np.append(flux_even, [flux[ii]], axis=0)

#check that new time_even is ~ regularly spaced
plt.plot(np.diff(time_even), color='orange')
plt.ylabel('Time diff [days]')

#we artificially make the time array perfectly regular for this project. See Lomb-Scargle course for dealing with non-regular data.
time =time_even#[0:1000]
time_res=(time[-1]-time[0])/len(time)
time = [time[0]+ii*time_res for ii in range(len(time))]
flux =flux_even#[0:1000]
flux = np.nan_to_num(flux, copy=True, nan=1, posinf=None, neginf=None)

#plot new lightcurve
plt.figure()
plt.plot(time, flux, color='dodgerblue')
plt.ylabel('Time [days]')
plt.ylabel('Flux')

#we artificially make the time array perfectly regular for this project. See Lomb-Scargle course for dealing with non-regular data.
time =time_even
time_res=(time[-1]-time[0])/len(time)
time = [time[0]+ii*time_res for ii in range(len(time))]
flux =flux_even
flux = np.nan_to_num(flux, copy=True, nan=1, posinf=None, neginf=None)

#plot new lightcurve
plt.figure()
plt.plot(time, flux, color='crimson')
plt.ylabel('Time [days]')
plt.ylabel('Flux')

# Number of sample points
N = len(time)
# sample spacing
T = time[1]-time[0]

#compute FFT (or PSD...)
yf = np.fft.fft(flux)
xf = np.fft.fftfreq(N, T)[1:N//2]

fig, ax = plt.subplots(1,1, figsize=(9,4))
ax.set_yscale('log')
ax.plot(xf, np.abs(yf[1:N//2]), color='crimson')

ax.set_xlabel(r'Frequency [day$^{-1}$]')
ax.set_ylabel(r'FFT Amplitude')

#Then measure numax and delta nu
#Then compiute Mass and Radius estimates

```

```python
nuMaxArray = xf[np.argsort(np.abs(yf[1:N//2]))[-10:]]
nuMaxArray = nuMaxArray[np.argsort(nuMaxArray)[-3:]]
print(np.diff(nuMaxArray))
delta_nu = nuMaxArray[2]-nuMaxArray[0]
nu_max = nuMaxArray[1]
print(delta_nu)
print(nu_max)

muHz2perday = 10**6/(24*60*60)

mass = (nu_max/(3090*muHz2perday))**3 * (delta_nu/(135.1*muHz2perday))**(-4) * (5800/4881)**1.5
print(mass)

radius = (nu_max/(3090*muHz2perday))**1 * (delta_nu/(135.1*muHz2perday))**(-2) * (5800/4881)**0.5
print(radius)
```


# Computational Physics 2: Statistical analysis

**Morten Hjorth-Jensen  Email morten.hjorth-jensen@fys.uio.no**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

Date: **Mar 5, 2020**

Copyright 1999-2020, Morten Hjorth-Jensen  Email morten.hjorth-jensen@fys.uio.no. Released under CC Attribution-NonCommercial 4.0 license

## Resampling methods
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.

## Resampling approaches can be computationally expensive
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model’s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.

## Why resampling methods ?
**Statistical analysis.**

    * Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods

    * The results can be analysed with the same statistical tools as we would use analysing experimental data.

    * As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.

## Statistical analysis
    * As in other experiments, many numerical  experiments have two classes of errors:

      * Statistical errors

      * Systematical errors

    * Statistical errors can be estimated using standard tools from statistics

    * Systematical errors are method specific and must be treated differently from case to case.

## Statistics
The *probability distribution function (PDF)* is a function
$p(x)$ on the domain which, in the discrete case, gives us the
probability or relative frequency with which these values of $X$ occur:

$$
p(x) = \mathrm{prob}(X=x)
$$

In the continuous case, the PDF does not directly depict the
actual probability. Instead we define the probability for the
stochastic variable to assume any value on an infinitesimal interval
around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us
the *density* of the probability rather than the probability
itself. The probability for a stochastic variable to assume any value
on a non-infinitesimal interval $[a,\,b]$ is then just the integral:

$$
\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx
$$

Qualitatively speaking, a stochastic variable represents the values of
numbers chosen as if by chance from some specified PDF so that the
selection of a large set of these numbers reproduces this PDF.

## Statistics, moments
A particularly useful class of special expectation values are the
*moments*. The $n$-th moment of the PDF $p$ is defined as
follows:

$$
\langle x^n\rangle \equiv \int\! x^n p(x)\,dx
$$

The zero-th moment $\langle 1\rangle$ is just the normalization condition of
$p$. The first moment, $\langle x\rangle$, is called the *mean* of $p$
and often denoted by the letter $\mu$:

$$
\langle x\rangle = \mu \equiv \int\! x p(x)\,dx
$$

## Statistics, central moments
A special version of the moments is the set of *central moments*,
the n-th central moment defined as:

$$
\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx
$$

The zero-th and first central moments are both trivial, equal $1$ and
$0$, respectively. But the second central moment, known as the
*variance* of $p$, is of particular interest. For the stochastic
variable $X$, the variance is denoted as $\sigma^2_X$ or $\mathrm{var}(X)$:

$$
\begin{equation}
\sigma^2_X\ \ =\ \ \mathrm{var}(X)  =  \langle (x-\langle x\rangle)^2\rangle =
\int\! (x-\langle x\rangle)^2 p(x)\,dx
\label{_auto1} \tag{1}
\end{equation}
$$

$$
\begin{equation}
 =  \int\! \left(x^2 - 2 x \langle x\rangle^{2} +
  \langle x\rangle^2\right)p(x)\,dx
\label{_auto2} \tag{2}
\end{equation}
$$

$$
\begin{equation}
 =  \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2
\label{_auto3} \tag{3}
\end{equation}
$$

$$
\begin{equation}
 =  \langle x^2\rangle - \langle x\rangle^2
\label{_auto4} \tag{4}
\end{equation}
$$

The square root of the variance, $\sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle}$ is called the *standard deviation* of $p$. It is clearly just the RMS (root-mean-square)
value of the deviation of the PDF from its mean value, interpreted
qualitatively as the *spread* of $p$ around its mean.

## Statistics, covariance
Another important quantity is the so called covariance, a variant of
the above defined variance. Consider again the set $\{X_i\}$ of $n$
stochastic variables (not necessarily uncorrelated) with the
multivariate PDF $P(x_1,\dots,x_n)$. The *covariance* of two
of the stochastic variables, $X_i$ and $X_j$, is defined as follows:

$$
\mathrm{cov}(X_i,\,X_j) \equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle
\nonumber
$$

$$
\begin{equation}
=
\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\,
P(x_1,\dots,x_n)\,dx_1\dots dx_n
\label{eq:def_covariance} \tag{5}
\end{equation}
$$

with

$$
\langle x_i\rangle =
\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n
$$

## Statistics, more covariance
If we consider the above covariance as a matrix $C_{ij}=\mathrm{cov}(X_i,\,X_j)$, then the diagonal elements are just the familiar
variances, $C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i)$. It turns out that
all the off-diagonal elements are zero if the stochastic variables are
uncorrelated. This is easy to show, keeping in mind the linearity of
the expectation value. Consider the stochastic variables $X_i$ and
$X_j$, ($i\neq j$):

$$
\begin{equation}
\mathrm{cov}(X_i,\,X_j) = \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle
\label{_auto5} \tag{6}
\end{equation}
$$

$$
\begin{equation}
=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle
\label{_auto6} \tag{7}
\end{equation}
$$

$$
\begin{equation}
=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle +
\langle \langle x_i\rangle\langle x_j\rangle\rangle
\label{_auto7} \tag{8}
\end{equation}
$$

$$
\begin{equation}
=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle +
\langle x_i\rangle\langle x_j\rangle
\label{_auto8} \tag{9}
\end{equation}
$$

$$
\begin{equation}
=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle
\label{_auto9} \tag{10}
\end{equation}
$$

## Statistics, independent variables
If $X_i$ and $X_j$ are independent, we get
$\langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle$, resulting in $\mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j)$.

Also useful for us is the covariance of linear combinations of
stochastic variables. Let $\{X_i\}$ and $\{Y_i\}$ be two sets of
stochastic variables. Let also $\{a_i\}$ and $\{b_i\}$ be two sets of
scalars. Consider the linear combination:

$$
U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j
$$

By the linearity of the expectation value

$$
\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j)
$$

## Statistics, more variance
Now, since the variance is just $\mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i)$, we get
the variance of the linear combination $U = \sum_i a_i X_i$:

$$
\begin{equation}
\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j)
\label{eq:variance_linear_combination} \tag{11}
\end{equation}
$$

And in the special case when the stochastic variables are
uncorrelated, the off-diagonal elements of the covariance are as we
know zero, resulting in:

2
0

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i)
$$

which will become very useful in our study of the error in the mean
value of a set of measurements.

## Statistics and stochastic processes
A *stochastic process* is a process that produces sequentially a
chain of values:

$$
\{x_1, x_2,\dots\,x_k,\dots\}.
$$

We will call these
values our *measurements* and the entire set as our measured
*sample*.  The action of measuring all the elements of a sample
we will call a stochastic *experiment* since, operationally,
they are often associated with results of empirical observation of
some physical or mathematical phenomena; precisely an experiment. We
assume that these values are distributed according to some
PDF $p_X^{\phantom X}(x)$, where $X$ is just the formal symbol for the
stochastic variable whose PDF is $p_X^{\phantom X}(x)$. Instead of
trying to determine the full distribution $p$ we are often only
interested in finding the few lowest moments, like the mean
$\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$.

## Statistics and sample variables
In practical situations a sample is always of finite size. Let that
size be $n$. The expectation value of a sample, the *sample mean*, is then defined as follows:

$$
\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k
$$

The *sample variance* is:

$$
\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2
$$

its square root being the *standard deviation of the sample*. The
*sample covariance* is:

$$
\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n)
$$

## Statistics, sample variance and covariance
Note that the sample variance is the sample covariance without the
cross terms. In a similar manner as the covariance in Eq. ([5](#eq:def_covariance)) is a measure of the correlation between
two stochastic variables, the above defined sample covariance is a
measure of the sequential correlation between succeeding measurements
of a sample.

These quantities, being known experimental values, differ
significantly from and must not be confused with the similarly named
quantities for stochastic variables, mean $\mu_X$, variance $\mathrm{var}(X)$
and covariance $\mathrm{cov}(X,Y)$.

## Statistics, law of large numbers
The law of large numbers
states that as the size of our sample grows to infinity, the sample
mean approaches the true mean $\mu_X^{\phantom X}$ of the chosen PDF:

$$
\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X}
$$

The sample mean $\bar{x}_n$ works therefore as an estimate of the true
mean $\mu_X^{\phantom X}$.

What we need to find out is how good an approximation $\bar{x}_n$ is to
$\mu_X^{\phantom X}$. In any stochastic measurement, an estimated
mean is of no use to us without a measure of its error. A quantity
that tells us how well we can reproduce it in another experiment. We
are therefore interested in the PDF of the sample mean itself. Its
standard deviation will be a measure of the spread of sample means,
and we will simply call it the *error* of the sample mean, or
just sample error, and denote it by $\mathrm{err}_X^{\phantom X}$. In
practice, we will only be able to produce an *estimate* of the
sample error since the exact value would require the knowledge of the
true PDFs behind, which we usually do not have.

## Statistics, more on sample error
Let us first take a look at what happens to the sample error as the
size of the sample grows. In a sample, each of the measurements $x_i$
can be associated with its own stochastic variable $X_i$. The
stochastic variable $\overline X_n$ for the sample mean $\bar{x}_n$ is
then just a linear combination, already familiar to us:

$$
\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i
$$

All the coefficients are just equal $1/n$. The PDF of $\overline X_n$,
denoted by $p_{\overline X_n}(x)$ is the desired PDF of the sample
means.

## Statistics
The probability density of obtaining a sample mean $\bar x_n$
is the product of probabilities of obtaining arbitrary values $x_1,
x_2,\dots,x_n$ with the constraint that the mean of the set $\{x_i\}$
is $\bar x_n$:

$$
p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots
\int p_X^{\phantom X}(x_n)\
\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1
$$

And in particular we are interested in its variance $\mathrm{var}(\overline X_n)$.

## Statistics, central limit theorem
It is generally not possible to express $p_{\overline X_n}(x)$ in a
closed form given an arbitrary PDF $p_X^{\phantom X}$ and a number
$n$. But for the limit $n\to\infty$ it is possible to make an
approximation. The very important result is called *the central limit theorem*. It tells us that as $n$ goes to infinity,
$p_{\overline X_n}(x)$ approaches a Gaussian distribution whose mean
and variance equal the true mean and variance, $\mu_{X}^{\phantom X}$
and $\sigma_{X}^{2}$, respectively:

$$
\begin{equation}
\lim_{n\to\infty} p_{\overline X_n}(x) =
\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2}
e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}}
\label{eq:central_limit_gaussian} \tag{12}
\end{equation}
$$

## Statistics, more technicalities
The desired variance
$\mathrm{var}(\overline X_n)$, i.e. the sample error squared
$\mathrm{err}_X^2$, is given by:

$$
\begin{equation}
\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2}
\sum_{ij} \mathrm{cov}(X_i, X_j)
\label{eq:error_exact} \tag{13}
\end{equation}
$$

We see now that in order to calculate the exact error of the sample
with the above expression, we would need the true means
$\mu_{X_i}^{\phantom X}$ of the stochastic variables $X_i$. To
calculate these requires that we know the true multivariate PDF of all
the $X_i$. But this PDF is unknown to us, we have only got the measurements of
one sample. The best we can do is to let the sample itself be an
estimate of the PDF of each of the $X_i$, estimating all properties of
$X_i$ through the measurements of the sample.

## Statistics
Our estimate of $\mu_{X_i}^{\phantom X}$ is then the sample mean $\bar x$
itself, in accordance with the the central limit theorem:

$$
\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x
$$

Using $\bar x$ in place of $\mu_{X_i}^{\phantom X}$ we can give an
*estimate* of the covariance in Eq. ([13](#eq:error_exact))

$$
\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle
\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle,
$$

resulting in

$$
\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x)
$$

## Statistics and sample variance
By the same procedure we can use the sample variance as an
estimate of the variance of any of the stochastic variables $X_i$

$$
\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber,
$$

which is approximated as

$$
\begin{equation}
\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x)
\label{eq:var_estimate_i_think} \tag{14}
\end{equation}
$$

Now we can calculate an estimate of the error
$\mathrm{err}_X^{\phantom X}$ of the sample mean $\bar x_n$:

$$
\mathrm{err}_X^2
=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber
$$

$$
\approx\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber
$$

$$
\begin{equation}
=\frac{1}{n}\mathrm{cov}(x)
\label{eq:error_estimate} \tag{15}
\end{equation}
$$

which is nothing but the sample covariance divided by the number of
measurements in the sample.

## Statistics, uncorrelated results

In the special case that the measurements of the sample are
uncorrelated (equivalently the stochastic variables $X_i$ are
uncorrelated) we have that the off-diagonal elements of the covariance
are zero. This gives the following estimate of the sample error:

$$
\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) =
\frac{1}{n^2} \sum_i \mathrm{var}(X_i),
$$

resulting in

$$
\begin{equation}
\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x)
\label{eq:error_estimate_uncorrel} \tag{16}
\end{equation}
$$

where in the second step we have used Eq. ([14](#eq:var_estimate_i_think)).
The error of the sample is then just its standard deviation divided by
the square root of the number of measurements the sample contains.
This is a very useful formula which is easy to compute. It acts as a
first approximation to the error, but in numerical experiments, we
cannot overlook the always present correlations.

## Statistics, computations
For computational purposes one usually splits up the estimate of
$\mathrm{err}_X^2$, given by Eq. ([15](#eq:error_estimate)), into two
parts

$$
\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)),
$$

which equals

$$
\begin{equation}
\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k<l} (x_k - \bar x_n)(x_l - \bar x_n)
\label{eq:error_estimate_split_up} \tag{17}
\end{equation}
$$

The first term is the same as the error in the uncorrelated case,
Eq. ([16](#eq:error_estimate_uncorrel)). This means that the second
term accounts for the error correction due to correlation between the
measurements. For uncorrelated measurements this second term is zero.

## Statistics, more on computations of errors
Computationally the uncorrelated first term is much easier to treat
efficiently than the second.

$$
\mathrm{var}(x) = \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)^2 =
\left(\frac{1}{n}\sum_{k=1}^n x_k^2\right) - \bar x_n^2
$$

We just accumulate separately the values $x^2$ and $x$ for every
measurement $x$ we receive. The correlation term, though, has to be
calculated at the end of the experiment since we need all the
measurements to calculate the cross terms. Therefore, all measurements
have to be stored throughout the experiment.

## Statistics, wrapping up 1
Let us analyze the problem by splitting up the correlation term into
partial sums of the form:

$$
f_d = \frac{1}{n-d}\sum_{k=1}^{n-d}(x_k - \bar x_n)(x_{k+d} - \bar x_n)
$$

The correlation term of the error can now be rewritten in terms of
$f_d$

$$
\frac{2}{n}\sum_{k<l} (x_k - \bar x_n)(x_l - \bar x_n) =
2\sum_{d=1}^{n-1} f_d
$$

The value of $f_d$ reflects the correlation between measurements
separated by the distance $d$ in the sample samples.  Notice that for
$d=0$, $f$ is just the sample variance, $\mathrm{var}(x)$. If we divide $f_d$
by $\mathrm{var}(x)$, we arrive at the so called *autocorrelation function*

$$
\kappa_d = \frac{f_d}{\mathrm{var}(x)}
$$

which gives us a useful measure of pairwise correlations
starting always at $1$ for $d=0$.

## Statistics, final expression
The sample error (see eq. ([17](#eq:error_estimate_split_up))) can now be
written in terms of the autocorrelation function:

$$
\mathrm{err}_X^2 =
\frac{1}{n}\mathrm{var}(x)+\frac{2}{n}\cdot\mathrm{var}(x)\sum_{d=1}^{n-1}
\frac{f_d}{\mathrm{var}(x)}\nonumber
$$

$$
=
\left(1+2\sum_{d=1}^{n-1}\kappa_d\right)\frac{1}{n}\mathrm{var}(x)\nonumber
$$

$$
\begin{equation}
=\frac{\tau}{n}\cdot\mathrm{var}(x)
\label{eq:error_estimate_corr_time} \tag{18}
\end{equation}
$$

and we see that $\mathrm{err}_X$ can be expressed in terms the
uncorrelated sample variance times a correction factor $\tau$ which
accounts for the correlation between measurements. We call this
correction factor the *autocorrelation time*:

$$
\begin{equation}
\tau = 1+2\sum_{d=1}^{n-1}\kappa_d
\label{eq:autocorrelation_time} \tag{19}
\end{equation}
$$

## Statistics, effective number of correlations
For a correlation free experiment, $\tau$
equals 1. From the point of view of
eq. ([18](#eq:error_estimate_corr_time)) we can interpret a sequential
correlation as an effective reduction of the number of measurements by
a factor $\tau$. The effective number of measurements becomes:

$$
n_\mathrm{eff} = \frac{n}{\tau}
$$

To neglect the autocorrelation time $\tau$ will always cause our
simple uncorrelated estimate of $\mathrm{err}_X^2\approx \mathrm{var}(x)/n$ to
be less than the true sample error. The estimate of the error will be
too *good*. On the other hand, the calculation of the full
autocorrelation time poses an efficiency problem if the set of
measurements is very large.

## Can we understand this? Time Auto-correlation Function
The so-called time-displacement autocorrelation $\phi(t)$ for a quantity $\mathbf{M}$ is given by

$$
\phi(t) = \int dt' \left[\mathbf{M}(t')-\langle \mathbf{M} \rangle\right]\left[\mathbf{M}(t'+t)-\langle \mathbf{M} \rangle\right],
$$

which can be rewritten as

$$
\phi(t) = \int dt' \left[\mathbf{M}(t')\mathbf{M}(t'+t)-\langle \mathbf{M} \rangle^2\right],
$$

where $\langle \mathbf{M} \rangle$ is the average value and
$\mathbf{M}(t)$ its instantaneous value. We can discretize this function as follows, where we used our
set of computed values $\mathbf{M}(t)$ for a set of discretized times (our Monte Carlo cycles corresponding to moving all electrons?)

$$
\phi(t)  = \frac{1}{t_{\mathrm{max}}-t}\sum_{t'=0}^{t_{\mathrm{max}}-t}\mathbf{M}(t')\mathbf{M}(t'+t)
-\frac{1}{t_{\mathrm{max}}-t}\sum_{t'=0}^{t_{\mathrm{max}}-t}\mathbf{M}(t')\times
\frac{1}{t_{\mathrm{max}}-t}\sum_{t'=0}^{t_{\mathrm{max}}-t}\mathbf{M}(t'+t).
\label{eq:phitf} \tag{20}
$$

## Time Auto-correlation Function
One should be careful with times close to $t_{\mathrm{max}}$, the upper limit of the sums
becomes small and we end up integrating over a rather small time interval. This means that the statistical
error in $\phi(t)$ due to the random nature of the fluctuations in $\mathbf{M}(t)$ can become large.

One should therefore choose $t \ll t_{\mathrm{max}}$.

Note that the variable $\mathbf{M}$ can be any expectation values of interest.

The time-correlation function gives a measure of the correlation between the various values of the variable
at a time $t'$ and a time $t'+t$. If we multiply the values of $\mathbf{M}$ at these two different times,
we will get a positive contribution if they are fluctuating in the same direction, or a negative value
if they fluctuate in the opposite direction. If we then integrate over time, or use the discretized version of, the time correlation function $\phi(t)$ should take a non-zero value if the fluctuations are
correlated, else it should gradually go to zero. For times a long way apart
the different values of $\mathbf{M}$  are most likely
uncorrelated and $\phi(t)$ should be zero.

## Time Auto-correlation Function
We can derive the correlation time by observing that our Metropolis algorithm is based on a random
walk in the space of all  possible spin configurations.
Our probability
distribution function $\mathbf{\hat{w}}(t)$ after a given number of time steps $t$ could be written as

$$
\mathbf{\hat{w}}(t) = \mathbf{\hat{W}^t\hat{w}}(0),
$$

with $\mathbf{\hat{w}}(0)$ the distribution at $t=0$ and $\mathbf{\hat{W}}$ representing the
transition probability matrix.
We can always expand $\mathbf{\hat{w}}(0)$ in terms of the right eigenvectors of
$\mathbf{\hat{v}}$ of $\mathbf{\hat{W}}$ as

$$
\mathbf{\hat{w}}(0)  = \sum_i\alpha_i\mathbf{\hat{v}}_i,
$$

resulting in

$$
\mathbf{\hat{w}}(t) = \mathbf{\hat{W}}^t\mathbf{\hat{w}}(0)=\mathbf{\hat{W}}^t\sum_i\alpha_i\mathbf{\hat{v}}_i=
\sum_i\lambda_i^t\alpha_i\mathbf{\hat{v}}_i,
$$

with $\lambda_i$ the $i^{\mathrm{th}}$ eigenvalue corresponding to
the eigenvector $\mathbf{\hat{v}}_i$.

## Time Auto-correlation Function
If we assume that $\lambda_0$ is the largest eigenvector we see that in the limit $t\rightarrow \infty$,
$\mathbf{\hat{w}}(t)$ becomes proportional to the corresponding eigenvector
$\mathbf{\hat{v}}_0$. This is our steady state or final distribution.

We can relate this property to an observable like the mean energy.
With the probabilty $\mathbf{\hat{w}}(t)$ (which in our case is the squared trial wave function) we
can write the expectation values as

$$
\langle \mathbf{M}(t) \rangle  = \sum_{\mu} \mathbf{\hat{w}}(t)_{\mu}\mathbf{M}_{\mu},
$$

or as the scalar of a  vector product

$$
\langle \mathbf{M}(t) \rangle  = \mathbf{\hat{w}}(t)\mathbf{m},
$$

with $\mathbf{m}$ being the vector whose elements are the values of $\mathbf{M}_{\mu}$ in its
various microstates $\mu$.

## Time Auto-correlation Function
We rewrite this relation  as

$$
\langle \mathbf{M}(t) \rangle  = \mathbf{\hat{w}}(t)\mathbf{m}=\sum_i\lambda_i^t\alpha_i\mathbf{\hat{v}}_i\mathbf{m}_i.
$$

If we define $m_i=\mathbf{\hat{v}}_i\mathbf{m}_i$ as the expectation value of
$\mathbf{M}$ in the $i^{\mathrm{th}}$ eigenstate we can rewrite the last equation as

$$
\langle \mathbf{M}(t) \rangle  = \sum_i\lambda_i^t\alpha_im_i.
$$

Since we have that in the limit $t\rightarrow \infty$ the mean value is dominated by the
the largest eigenvalue $\lambda_0$, we can rewrite the last equation as

$$
\langle \mathbf{M}(t) \rangle  = \langle \mathbf{M}(\infty) \rangle+\sum_{i\ne 0}\lambda_i^t\alpha_im_i.
$$

We define the quantity

$$
\tau_i=-\frac{1}{log\lambda_i},
$$

and rewrite the last expectation value as

$$
\langle \mathbf{M}(t) \rangle  = \langle \mathbf{M}(\infty) \rangle+\sum_{i\ne 0}\alpha_im_ie^{-t/\tau_i}.
\label{eq:finalmeanm} \tag{21}
$$

## Time Auto-correlation Function

The quantities $\tau_i$ are the correlation times for the system. They control also the auto-correlation function
discussed above.  The longest correlation time is obviously given by the second largest
eigenvalue $\tau_1$, which normally defines the correlation time discussed above. For large times, this is the
only correlation time that survives. If higher eigenvalues of the transition matrix are well separated from
$\lambda_1$ and we simulate long enough,  $\tau_1$ may well define the correlation time.
In other cases we may not be able to extract a reliable result for $\tau_1$.
Coming back to the time correlation function $\phi(t)$ we can present a more general definition in terms
of the mean magnetizations $ \langle \mathbf{M}(t) \rangle$. Recalling that the mean value is equal
to $ \langle \mathbf{M}(\infty) \rangle$ we arrive at the expectation values

$$
\phi(t) =\langle \mathbf{M}(0)-\mathbf{M}(\infty)\rangle \langle \mathbf{M}(t)-\mathbf{M}(\infty)\rangle,
$$

resulting in

$$
\phi(t) =\sum_{i,j\ne 0}m_i\alpha_im_j\alpha_je^{-t/\tau_i},
$$

which is appropriate for all times.

## Correlation Time

If the correlation function decays exponentially

$$
\phi (t) \sim \exp{(-t/\tau)}
$$

then the exponential correlation time can be computed as the average

$$
\tau_{\mathrm{exp}}  =  -\langle  \frac{t}{log|\frac{\phi(t)}{\phi(0)}|} \rangle.
$$

If the decay is exponential, then

$$
\int_0^{\infty} dt \phi(t)  = \int_0^{\infty} dt \phi(0)\exp{(-t/\tau)}  = \tau \phi(0),
$$

which  suggests another measure of correlation

$$
\tau_{\mathrm{int}} = \sum_k \frac{\phi(k)}{\phi(0)},
$$

called the integrated correlation time.

## Resampling methods: Jackknife and Bootstrap

Two famous
resampling methods are the **independent bootstrap** and **the jackknife**.

The jackknife is a special case of the independent bootstrap. Still, the jackknife was made
popular prior to the independent bootstrap. And as the popularity of
the independent bootstrap soared, new variants, such as **the dependent bootstrap**.

The Jackknife and independent bootstrap work for
independent, identically distributed random variables.
If these conditions are not
satisfied, the methods will fail.  Yet, it should be said that if the data are
independent, identically distributed, and we only want to estimate the
variance of $\overline{X}$ (which often is the case), then there is no
need for bootstrapping.

## Resampling methods: Jackknife

The Jackknife works by making many replicas of the estimator $\widehat{\theta}$.
The jackknife is a resampling method, we explained that this happens by scrambling the data in some way. When using the jackknife, this is done by systematically leaving out one observation from the vector of observed values $\hat{x} = (x_1,x_2,\cdots,X_n)$.
Let $\hat{x}_i$ denote the vector

$$
\hat{x}_i = (x_1,x_2,\cdots,x_{i-1},x_{i+1},\cdots,x_n),
$$

which equals the vector $\hat{x}$ with the exception that observation
number $i$ is left out. Using this notation, define
$\widehat{\theta}_i$ to be the estimator
$\widehat{\theta}$ computed using $\vec{X}_i$.

## Resampling methods: Jackknife estimator

To get an estimate for the bias and
standard error of $\widehat{\theta}$, use the following
estimators for each component of $\widehat{\theta}$

$$
\widehat{\mathrm{Bias}}(\widehat \theta,\theta) = (n-1)\left( - \widehat{\theta} + \frac{1}{n}\sum_{i=1}^{n} \widehat \theta_i \right) \qquad \text{and} \qquad \widehat{\sigma}^2_{\widehat{\theta} } = \frac{n-1}{n}\sum_{i=1}^{n}( \widehat{\theta}_i - \frac{1}{n}\sum_{j=1}^{n}\widehat \theta_j )^2.
$$

## Jackknife code example

```
from numpy import *
from numpy.random import randint, randn
from time import time

def jackknife(data, stat):
    n = len(data);t = zeros(n); inds = arange(n); t0 = time()
    ## 'jackknifing' by leaving out an observation for each i
    for i in range(n):
        t[i] = stat(delete(data,i) )

    # analysis
    print("Runtime: %g sec" % (time()-t0)); print("Jackknife Statistics :")
    print("original           bias      std. error")
    print("%8g %14g %15g" % (stat(data),(n-1)*mean(t)/n, (n*var(t))**.5))

    return t

# Returns mean of data samples
def stat(data):
    return mean(data)

mu, sigma = 100, 15
datapoints = 10000
x = mu + sigma*random.randn(datapoints)
# jackknife returns the data sample
t = jackknife(x, stat)
```

## Resampling methods: Bootstrap
Bootstrapping is a nonparametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. Bootstrapping offers a number of
advantages:
1. The bootstrap is quite general, although there are some cases in which it fails.

2. Because it does not require distributional assumptions (such as normally distributed errors), the bootstrap can provide more accurate inferences when the data are not well behaved or when the sample size is small.

3. It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.

4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).

## Resampling methods: Bootstrap background

Since $\widehat{\theta} = \widehat{\theta}(\hat{X})$ is a function of random variables,
$\widehat{\theta}$ itself must be a random variable. Thus it has
a pdf, call this function $p(\hat{t})$. The aim of the bootstrap is to
estimate $p(\hat{t})$ by the relative frequency of
$\widehat{\theta}$. You can think of this as using a histogram
in the place of $p(\hat{t})$. If the relative frequency closely
resembles $p(\vec{t})$, then using numerics, it is straight forward to
estimate all the interesting parameters of $p(\hat{t})$ using point
estimators.

## Resampling methods: More Bootstrap background

In the case that $\widehat{\theta}$ has
more than one component, and the components are independent, we use the
same estimator on each component separately.  If the probability
density function of $X_i$, $p(x)$, had been known, then it would have
been straight forward to do this by:
1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$.

2. Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$.

By repeated use of (1) and (2), many
estimates of $\widehat{\theta}$ could have been obtained. The
idea is to use the relative frequency of $\widehat{\theta}^*$
(think of a histogram) as an estimate of $p(\hat{t})$.

## Resampling methods: Bootstrap approach

But
unless there is enough information available about the process that
generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general
unknown. Therefore, [Efron in 1979](https://projecteuclid.org/euclid.aos/1176344552)  asked the
question: What if we replace $p(x)$ by the relative frequency
of the observation $X_i$; if we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.

Instead of generating the histogram for the relative
frequency of the observation $X_i$, just draw the values
$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector
$\hat{X}$.

## Resampling methods: Bootstrap steps

The independent bootstrap works like this:

1. Draw with replacement $n$ numbers for the observed variables $\hat{x} = (x_1,x_2,\cdots,x_n)$.

2. Define a vector $\hat{x}^*$ containing the values which were drawn from $\hat{x}$.

3. Using the vector $\hat{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\hat{x}^*$.

4. Repeat this process $k$ times.

When you are done, you can draw a histogram of the relative frequency of $\widehat \theta^*$. This is your estimate of the probability distribution $p(t)$. Using this probability distribution you can estimate any statistics thereof. In principle you never draw the histogram of the relative frequency of $\widehat{\theta}^*$. Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of $\widehat \theta$, apply the etsimator $\widehat \sigma^2$ to the values $\widehat \theta ^*$.

## Code example for the Bootstrap method
The following code starts with a Gaussian distribution with mean value $\mu =100$ and variance $\sigma=15$. We use this to generate the data used in the bootstrap analysis. The bootstrap analysis returns a data set after a given number of bootstrap operations (as many as we have data points). This data set consists of estimated mean values for each bootstrap operation. The histogram generated by the bootstrap method shows that the distribution for these mean values is also a Gaussian, centered around the mean value $\mu=100$ but with standard deviation $\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in this case the same as the number of original data points). The value of the standard deviation is what we expect from the central limit theorem.

```
%matplotlib inline

from numpy import *
from numpy.random import randint, randn
from time import time
import matplotlib.mlab as mlab
import matplotlib.pyplot as plt

# Returns mean of bootstrap samples
def stat(data):
    return mean(data)

# Bootstrap algorithm
def bootstrap(data, statistic, R):
    t = zeros(R); n = len(data); inds = arange(n); t0 = time()

    # non-parametric bootstrap
    for i in range(R):
        t[i] = statistic(data[randint(0,n,n)])

    # analysis
    print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
    print("original           bias      std. error")
    print("%8g %8g %14g %15g" % (statistic(data), std(data),\
                             mean(t), \
                             std(t)))
    return t

mu, sigma = 100, 15
datapoints = 10000
x = mu + sigma*random.randn(datapoints)
# bootstrap returns the data sample                                                                                t = bootstrap(x, stat, datapoints)
# the histogram of the bootstrapped  data                                                                                n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)

# add a 'best fit' line
y = mlab.normpdf( binsboot, mean(t), std(t))
lt = plt.plot(binsboot, y, 'r--', linewidth=1)
plt.xlabel('Smarts')
plt.ylabel('Probability')
plt.axis([99.5, 100.6, 0, 3.0])
plt.grid(True)

plt.show()
```

## Resampling methods: Blocking

The blocking method was made popular by [Flyvbjerg and Pedersen (1989)](https://aip.scitation.org/doi/10.1063/1.457480)
and has become one of the standard ways to estimate
$V(\widehat{\theta})$ for exactly one $\widehat{\theta}$, namely
$\widehat{\theta} = \overline{X}$.

Assume $n = 2^d$ for some integer $d>1$ and $X_1,X_2,\cdots, X_n$ is a stationary time series to begin with.
Moreover, assume that the time series is asymptotically uncorrelated. We switch to vector notation by arranging $X_1,X_2,\cdots,X_n$ in an $n$-tuple. Define:

$$
\begin{align*}
\hat{X} = (X_1,X_2,\cdots,X_n).
\end{align*}
$$

The strength of the blocking method is when the number of
observations, $n$ is large. For large $n$, the complexity of dependent
bootstrapping scales poorly, but the blocking method does not,
moreover, it becomes more accurate the larger $n$ is.

## Blocking Transformations
 We now define
blocking transformations. The idea is to take the mean of subsequent
pair of elements from $\vec{X}$ and form a new vector
$\vec{X}_1$. Continuing in the same way by taking the mean of
subsequent pairs of elements of $\vec{X}_1$ we obtain $\vec{X}_2$, and
so on.
Define $\vec{X}_i$ recursively by:

$$
(\vec{X}_0)_k \equiv (\vec{X})_k \nonumber
$$

$$
\begin{equation}
(\vec{X}_{i+1})_k \equiv \frac{1}{2}\Big( (\vec{X}_i)_{2k-1} +
(\vec{X}_i)_{2k} \Big) \qquad \text{for all} \qquad 1 \leq i \leq d-1
\label{_auto10} \tag{22}
\end{equation}
$$

The quantity $\vec{X}_k$ is
subject to $k$ **blocking transformations**.  We now have $d$ vectors
$\vec{X}_0, \vec{X}_1,\cdots,\vec X_{d-1}$ containing the subsequent
averages of observations. It turns out that if the components of
$\vec{X}$ is a stationary time series, then the components of
$\vec{X}_i$ is a stationary time series for all $0 \leq i \leq d-1$

We can then compute the autocovariance, the variance, sample mean, and
number of observations for each $i$.
Let $\gamma_i, \sigma_i^2,
\overline{X}_i$ denote the autocovariance, variance and average of the
elements of $\vec{X}_i$ and let $n_i$ be the number of elements of
$\vec{X}_i$. It follows by induction that $n_i = n/2^i$.

## Blocking Transformations

Using the
definition of the blocking transformation and the distributive
property of the covariance, it is clear that since $h =|i-j|$
we can define

$$
\gamma_{k+1}(h) = cov\left( ({X}_{k+1})_{i}, ({X}_{k+1})_{j} \right) \nonumber
$$

$$
=  \frac{1}{4}cov\left( ({X}_{k})_{2i-1} + ({X}_{k})_{2i}, ({X}_{k})_{2j-1} + ({X}_{k})_{2j} \right) \nonumber
$$

$$
\begin{equation}
=  \frac{1}{2}\gamma_{k}(2h) + \frac{1}{2}\gamma_k(2h+1) \hspace{0.1cm} \mathrm{h = 0}
\label{_auto11} \tag{23}
\end{equation}
$$

$$
\begin{equation}
=\frac{1}{4}\gamma_k(2h-1) + \frac{1}{2}\gamma_k(2h) + \frac{1}{4}\gamma_k(2h+1) \quad \mathrm{else}
\label{_auto12} \tag{24}
\end{equation}
$$

The quantity $\hat{X}$ is asymptotic uncorrelated by assumption, $\hat{X}_k$ is also asymptotic uncorrelated. Let's turn our attention to the variance of the sample mean $V(\overline{X})$.

## Blocking Transformations, getting there
We have

$$
\begin{equation}
V(\overline{X}_k) = \frac{\sigma_k^2}{n_k} + \underbrace{\frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h)}_{\equiv e_k} = \frac{\sigma^2_k}{n_k} + e_k \quad \text{if} \quad \gamma_k(0) = \sigma_k^2.
\label{_auto13} \tag{25}
\end{equation}
$$

The term $e_k$ is called the **truncation error**:

$$
\begin{equation}
e_k = \frac{2}{n_k} \sum_{h=1}^{n_k-1}\left( 1 - \frac{h}{n_k} \right)\gamma_k(h).
\label{_auto14} \tag{26}
\end{equation}
$$

We can show that $V(\overline{X}_i) = V(\overline{X}_j)$ for all $0 \leq i \leq d-1$ and $0 \leq j \leq d-1$.

## Blocking Transformations, final expressions

We can then wrap up

$$
n_{j+1} \overline{X}_{j+1}  = \sum_{i=1}^{n_{j+1}} (\hat{X}_{j+1})_i =  \frac{1}{2}\sum_{i=1}^{n_{j}/2} (\hat{X}_{j})_{2i-1} + (\hat{X}_{j})_{2i} \nonumber
$$

$$
\begin{equation}
= \frac{1}{2}\left[ (\hat{X}_j)_1 + (\hat{X}_j)_2 + \cdots + (\hat{X}_j)_{n_j} \right] = \underbrace{\frac{n_j}{2}}_{=n_{j+1}} \overline{X}_j = n_{j+1}\overline{X}_j.
\label{_auto15} \tag{27}
\end{equation}
$$

By repeated use of this equation we get $V(\overline{X}_i) = V(\overline{X}_0) = V(\overline{X})$ for all $0 \leq i \leq d-1$. This has the consequence that

$$
\begin{equation}
V(\overline{X}) = \frac{\sigma_k^2}{n_k} + e_k \qquad \text{for all} \qquad 0 \leq k \leq d-1. \label{eq:convergence} \tag{28}
\end{equation}
$$

Flyvbjerg and Petersen demonstrated that the sequence
$\{e_k\}_{k=0}^{d-1}$ is decreasing, and conjecture that the term
$e_k$ can be made as small as we would like by making $k$ (and hence
$d$) sufficiently large. The sequence is decreasing (Master of Science thesis by Marius Jonsson, UiO 2018).
It means we can apply blocking transformations until
$e_k$ is sufficiently small, and then estimate $V(\overline{X})$ by
$\widehat{\sigma}^2_k/n_k$.

For an elegant solution and proof of the blocking method, see the recent article of [Marius Jonsson (former MSc student of the Computational Physics group)](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.98.043304).

## Resampling analysis

Here we have tailored the codes to the output file from a VMC calculation with two electrons only interacting via a Coulomb interaction. We present first the bootstrap and blocking resampling with non-iid stochastic event.

```
# Common imports
import os

# Where to save the figures and data files
DATA_ID = "Results/EnergyMin"

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

infile = open(data_path("Energies.dat"),'r')

from numpy import std, mean, concatenate, arange, loadtxt, zeros, ceil
from numpy.random import randint
from time import time

def tsboot(data,statistic,R,l):
    t = zeros(R); n = len(data); k = int(ceil(float(n)/l));
    inds = arange(n); t0 = time()

    # time series bootstrap
    for i in range(R):
        # construct bootstrap sample from
        # k chunks of data. The chunksize is l
        _data = concatenate([data[j:j+l] for j in randint(0,n-l,k)])[0:n];
        t[i] = statistic(_data)

    # analysis
    print ("Runtime: %g sec" % (time()-t0)); print ("Bootstrap Statistics :")
    print ("original           bias      std. error")
    print ("%8g %14g %15g" % (statistic(data), \
                             mean(t) - statistic(data), \
                             std(t) ))
    return t
# Read in data
X = loadtxt(infile)
# statistic to be estimated. Takes two args.
# arg1: the data
def stat(data):
    return mean(data)
t = tsboot(X, stat, 2**12, 2**10)
```

The blocking code, based on the article of [Marius Jonsson](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.98.043304) is given here

```
# Common imports
import os

# Where to save the figures and data files
DATA_ID = "Results/EnergyMin"

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

infile = open(data_path("Energies.dat"),'r')

from numpy import log2, zeros, mean, var, sum, loadtxt, arange, array, cumsum, dot, transpose, diagonal, sqrt
from numpy.linalg import inv

def block(x):
    # preliminaries
    n = len(x)
    d = int(log2(n))
    s, gamma = zeros(d), zeros(d)
    mu = mean(x)

    # estimate the auto-covariance and variances
    # for each blocking transformation
    for i in arange(0,d):
        n = len(x)
        # estimate autocovariance of x
        gamma[i] = (n)**(-1)*sum( (x[0:(n-1)]-mu)*(x[1:n]-mu) )
        # estimate variance of x
        s[i] = var(x)
        # perform blocking transformation
        x = 0.5*(x[0::2] + x[1::2])

    # generate the test observator M_k from the theorem
    M = (cumsum( ((gamma/s)**2*2**arange(1,d+1)[::-1])[::-1] )  )[::-1]

    # we need a list of magic numbers
    q =array([6.634897,9.210340, 11.344867, 13.276704, 15.086272, 16.811894, 18.475307, 20.090235, 21.665994, 23.209251, 24.724970, 26.216967, 27.688250, 29.141238, 30.577914, 31.999927, 33.408664, 34.805306, 36.190869, 37.566235, 38.932173, 40.289360, 41.638398, 42.979820, 44.314105, 45.641683, 46.962942, 48.278236, 49.587884, 50.892181])

    # use magic to determine when we should have stopped blocking
    for k in arange(0,d):
        if(M[k] < q[k]):
            break
    if (k >= d-1):
        print("Warning: Use more data")
    return mu, s[k]/2**(d-k)

x = loadtxt(infile)
(mean, var) = block(x)
std = sqrt(var)
import pandas as pd
from pandas import DataFrame
data ={'Mean':[mean], 'STDev':[std]}
frame = pd.DataFrame(data,index=['Values'])
print(frame)
```


# GPyTorch Regression Tutorial (GPU)

(This notebook is the same as the [simple GP regression tutorial](../01_Exact_GPs/Simple_GP_Regression.ipynb) notebook, but does all computations on a GPU for acceleration.
Check out the [multi-GPU tutorial](./Simple_MultiGPU_GP_Regression.ipynb) if you have large datasets that needs multiple GPUs!)

## Introduction

In this notebook, we demonstrate many of the design features of GPyTorch using the simplest example, training an RBF kernel Gaussian process on a simple function. We'll be modeling the function

\begin{align}
  y &= \sin(2\pi x) + \epsilon \\
  \epsilon &\sim \mathcal{N}(0, 0.2)
\end{align}

with 11 training examples, and testing on 51 test examples.

```python
import math
import torch
import gpytorch
from matplotlib import pyplot as plt

%matplotlib inline
%load_ext autoreload
%autoreload 2
```

### Set up training data

In the next cell, we set up the training data for this example. We'll be using 11 regularly spaced points on [0,1] which we evaluate the function on and add Gaussian noise to get the training labels.

```python
# Training data is 11 points in [0,1] inclusive regularly spaced
train_x = torch.linspace(0, 1, 100)
# True function is sin(2*pi*x) with Gaussian noise
train_y = torch.sin(train_x * (2 * math.pi)) + torch.randn(train_x.size()) * 0.2
```

### Setting up the model

See [the simple GP regression tutorial](../01_Exact_GPs/Simple_GP_Regression.ipynb) for a detailed explanation for all the terms.

```python
class ExactGPModel(gpytorch.models.ExactGP):
    def __init__(self, train_x, train_y, likelihood):
        super(ExactGPModel, self).__init__(train_x, train_y, likelihood)
        self.mean_module = gpytorch.means.ConstantMean()
        self.covar_module = gpytorch.kernels.ScaleKernel(gpytorch.kernels.RBFKernel())

    def forward(self, x):
        mean_x = self.mean_module(x)
        covar_x = self.covar_module(x)
        return gpytorch.distributions.MultivariateNormal(mean_x, covar_x)

# initialize likelihood and model
likelihood = gpytorch.likelihoods.GaussianLikelihood()
model = ExactGPModel(train_x, train_y, likelihood)
```

## Using the GPU

To do computations on the GPU, we need to put our data and model onto the GPU. (This requires PyTorch with CUDA).

```python
train_x = train_x.cuda()
train_y = train_y.cuda()
model = model.cuda()
likelihood = likelihood.cuda()
```

That's it! All the training code is the same as in [the simple GP regression tutorial](../01_Exact_GPs/Simple_GP_Regression.ipynb).

### Training the model

```python
# Find optimal model hyperparameters
model.train()
likelihood.train()

# Use the adam optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.1)  # Includes GaussianLikelihood parameters

# "Loss" for GPs - the marginal log likelihood
mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)

training_iter = 50
for i in range(training_iter):
    # Zero gradients from previous iteration
    optimizer.zero_grad()
    # Output from model
    output = model(train_x)
    # Calc loss and backprop gradients
    loss = -mll(output, train_y)
    loss.backward()
    print('Iter %d/%d - Loss: %.3f   lengthscale: %.3f   noise: %.3f' % (
        i + 1, training_iter, loss.item(),
        model.covar_module.base_kernel.lengthscale.item(),
        model.likelihood.noise.item()
    ))
    optimizer.step()
```

    Iter 1/50 - Loss: 0.944   lengthscale: 0.693   noise: 0.693
    Iter 2/50 - Loss: 0.913   lengthscale: 0.644   noise: 0.644
    Iter 3/50 - Loss: 0.879   lengthscale: 0.598   noise: 0.598
    Iter 4/50 - Loss: 0.841   lengthscale: 0.555   noise: 0.554
    Iter 5/50 - Loss: 0.798   lengthscale: 0.514   noise: 0.513
    Iter 6/50 - Loss: 0.750   lengthscale: 0.475   noise: 0.474
    Iter 7/50 - Loss: 0.698   lengthscale: 0.439   noise: 0.437
    Iter 8/50 - Loss: 0.645   lengthscale: 0.405   noise: 0.402
    Iter 9/50 - Loss: 0.595   lengthscale: 0.372   noise: 0.369
    Iter 10/50 - Loss: 0.548   lengthscale: 0.342   noise: 0.339
    Iter 11/50 - Loss: 0.507   lengthscale: 0.315   noise: 0.310
    Iter 12/50 - Loss: 0.469   lengthscale: 0.292   noise: 0.284
    Iter 13/50 - Loss: 0.432   lengthscale: 0.272   noise: 0.259
    Iter 14/50 - Loss: 0.398   lengthscale: 0.255   noise: 0.236
    Iter 15/50 - Loss: 0.363   lengthscale: 0.241   noise: 0.215
    Iter 16/50 - Loss: 0.329   lengthscale: 0.230   noise: 0.196
    Iter 17/50 - Loss: 0.296   lengthscale: 0.222   noise: 0.178
    Iter 18/50 - Loss: 0.263   lengthscale: 0.215   noise: 0.162
    Iter 19/50 - Loss: 0.230   lengthscale: 0.210   noise: 0.147
    Iter 20/50 - Loss: 0.198   lengthscale: 0.207   noise: 0.134
    Iter 21/50 - Loss: 0.167   lengthscale: 0.205   noise: 0.122
    Iter 22/50 - Loss: 0.136   lengthscale: 0.205   noise: 0.110
    Iter 23/50 - Loss: 0.107   lengthscale: 0.206   noise: 0.100
    Iter 24/50 - Loss: 0.079   lengthscale: 0.208   noise: 0.091
    Iter 25/50 - Loss: 0.053   lengthscale: 0.211   noise: 0.083
    Iter 26/50 - Loss: 0.028   lengthscale: 0.215   noise: 0.076
    Iter 27/50 - Loss: 0.006   lengthscale: 0.220   noise: 0.069
    Iter 28/50 - Loss: -0.013   lengthscale: 0.225   noise: 0.063
    Iter 29/50 - Loss: -0.029   lengthscale: 0.231   noise: 0.058
    Iter 30/50 - Loss: -0.043   lengthscale: 0.237   noise: 0.053
    Iter 31/50 - Loss: -0.053   lengthscale: 0.243   noise: 0.049
    Iter 32/50 - Loss: -0.060   lengthscale: 0.249   noise: 0.045
    Iter 33/50 - Loss: -0.065   lengthscale: 0.254   noise: 0.042
    Iter 34/50 - Loss: -0.066   lengthscale: 0.259   noise: 0.039
    Iter 35/50 - Loss: -0.066   lengthscale: 0.262   noise: 0.037
    Iter 36/50 - Loss: -0.063   lengthscale: 0.265   noise: 0.035
    Iter 37/50 - Loss: -0.060   lengthscale: 0.266   noise: 0.033
    Iter 38/50 - Loss: -0.056   lengthscale: 0.266   noise: 0.032
    Iter 39/50 - Loss: -0.052   lengthscale: 0.265   noise: 0.031
    Iter 40/50 - Loss: -0.049   lengthscale: 0.262   noise: 0.030
    Iter 41/50 - Loss: -0.047   lengthscale: 0.259   noise: 0.029
    Iter 42/50 - Loss: -0.046   lengthscale: 0.254   noise: 0.029
    Iter 43/50 - Loss: -0.046   lengthscale: 0.249   noise: 0.029
    Iter 44/50 - Loss: -0.047   lengthscale: 0.243   noise: 0.029
    Iter 45/50 - Loss: -0.049   lengthscale: 0.237   noise: 0.029
    Iter 46/50 - Loss: -0.051   lengthscale: 0.231   noise: 0.029
    Iter 47/50 - Loss: -0.054   lengthscale: 0.225   noise: 0.030
    Iter 48/50 - Loss: -0.057   lengthscale: 0.219   noise: 0.030
    Iter 49/50 - Loss: -0.059   lengthscale: 0.214   noise: 0.031
    Iter 50/50 - Loss: -0.061   lengthscale: 0.210   noise: 0.032

### Make predictions with the model

First, we need to make some test data, and then throw it onto the GPU

```python
test_x = torch.linspace(0, 1, 51).cuda()
```

Now the rest of the code follows [the simple GP regression tutorial](../01_Exact_GPs/Simple_GP_Regression.ipynb).

```python
# Get into evaluation (predictive posterior) mode
model.eval()
likelihood.eval()

# Test points are regularly spaced along [0,1]
# Make predictions by feeding model through likelihood
with torch.no_grad(), gpytorch.settings.fast_pred_var():
    observed_pred = likelihood(model(test_x))
    mean = observed_pred.mean
    lower, upper = observed_pred.confidence_region()
```

For plotting, we're going to grab the data from the GPU and put it back on the CPU.
We can accomplish this with the `.cpu()` function.

```python
mean = mean.cpu()
lower = lower.cpu()
upper = upper.cpu()

train_x = train_x.cpu()
train_y = train_y.cpu()
test_x = test_x.cpu()
```

```python
with torch.no_grad():
    # Initialize plot
    f, ax = plt.subplots(1, 1, figsize=(4, 3))

    # Plot training data as black stars
    ax.plot(train_x.numpy(), train_y.numpy(), 'k*')
    # Plot predictive means as blue line
    ax.plot(test_x.numpy(), mean.numpy(), 'b')
    # Shade between the lower and upper confidence bounds
    ax.fill_between(test_x.numpy(), lower.numpy(), upper.numpy(), alpha=0.5)
    ax.set_ylim([-3, 3])
    ax.legend(['Observed Data', 'Mean', 'Confidence'])
```