# Optimised symbolic finite difference computation with Devito

Welcome to the interactive hands-on tutorial for [Devito](http://www.devitoproject.org). Devito is a domain-specific language (DSL) and code generation framework for the design of highly optimised finite difference kernels, and was primarily designed for use in seismic inversion methods. Devito utilises SymPy to allow the definition of matrix-free finite difference operators from high-level symbolic equations and generates optimised and automatically tuned code specific to a given target architecture.

This hands-on tutorial is intended to give you an initial flavour of the Devito framework and the power of symbolic computation. We will demonstrate how quickly explicit finite difference operators can be created from only a few lines of Python code, and how we can use them to implement complex imaging algorithms in very little time (literally!).

## Installation and setup

If you're seeing this on the projector, navigate to the following url to access the Azure notebooks project:

### TODO Add URL

Once here, clone the project to create your personal copy. Next, click on TODO to start the notebook container.

Now you should see a set of Jupyter notebooks, inlcuding this one called ```00_index.ipynb```. Open your copy of this notebook and proceed further.

```python
!pip install -r requirements.txt
```

    Collecting git+https://github.com/inducer/cgen (from -r requirements.txt (line 1))
      Cloning https://github.com/inducer/cgen to /tmp/pip-req-build-5ydpe78j
    Requirement already satisfied (use --upgrade to upgrade): cgen==2018.1 from git+https://github.com/inducer/cgen in /venv/lib/python3.6/site-packages (from -r requirements.txt (line 1))
    Collecting git+https://github.com/inducer/codepy (from -r requirements.txt (line 2))
      Cloning https://github.com/inducer/codepy to /tmp/pip-req-build-mpn21s66
    Requirement already satisfied (use --upgrade to upgrade): codepy==2017.2.2 from git+https://github.com/inducer/codepy in /venv/lib/python3.6/site-packages (from -r requirements.txt (line 2))
    Collecting git+https://github.com/opesci/devito (from -r requirements.txt (line 3))
      Cloning https://github.com/opesci/devito to /tmp/pip-req-build-axo_sfyj
    Requirement already satisfied: pytools>=2015.1.2 in /venv/lib/python3.6/site-packages (from cgen==2018.1->-r requirements.txt (line 1)) (2019.1)
    Requirement already satisfied: numpy>=1.6 in /venv/lib/python3.6/site-packages (from cgen==2018.1->-r requirements.txt (line 1)) (1.16.2)
    Requirement already satisfied: appdirs>=1.4.0 in /venv/lib/python3.6/site-packages (from codepy==2017.2.2->-r requirements.txt (line 2)) (1.4.3)
    Requirement already satisfied: six in /venv/lib/python3.6/site-packages (from codepy==2017.2.2->-r requirements.txt (line 2)) (1.12.0)
    Requirement already satisfied: sympy>=1.4 in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.4)
    Requirement already satisfied: scipy in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.2.1)
    Requirement already satisfied: pytest-runner in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (4.4)
    Requirement already satisfied: flake8>=2.1.0 in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (3.7.7)
    Requirement already satisfied: jedi in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.13.3)
    Requirement already satisfied: nbval in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.9.1)
    Requirement already satisfied: cached-property in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.5.1)
    Requirement already satisfied: psutil>=5.1.0 in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (5.6.1)
    Requirement already satisfied: py-cpuinfo in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (5.0.0)
    Requirement already satisfied: scikit-image in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.15.0)
    Requirement already satisfied: click in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (7.0)
    Requirement already satisfied: codecov in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.0.15)
    Requirement already satisfied: pytest-cov in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.6.1)
    Requirement already satisfied: multidict in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (4.5.2)
    Requirement already satisfied: frozendict in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.2)
    Requirement already satisfied: anytree>=2.4.3 in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.6.0)
    Requirement already satisfied: pyrevolve==1.0.2 in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.0.2)
    Requirement already satisfied: distributed>=1.21.8 in /venv/lib/python3.6/site-packages (from devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.27.0)
    Requirement already satisfied: decorator>=3.2.0 in /venv/lib/python3.6/site-packages (from pytools>=2015.1.2->cgen==2018.1->-r requirements.txt (line 1)) (4.4.0)
    Requirement already satisfied: mpmath>=0.19 in /venv/lib/python3.6/site-packages (from sympy>=1.4->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.1.0)
    Requirement already satisfied: pyflakes<2.2.0,>=2.1.0 in /venv/lib/python3.6/site-packages (from flake8>=2.1.0->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.1.1)
    Requirement already satisfied: pycodestyle<2.6.0,>=2.5.0 in /venv/lib/python3.6/site-packages (from flake8>=2.1.0->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.5.0)
    Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /venv/lib/python3.6/site-packages (from flake8>=2.1.0->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.6.1)
    Requirement already satisfied: entrypoints<0.4.0,>=0.3.0 in /venv/lib/python3.6/site-packages (from flake8>=2.1.0->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.3)
    Requirement already satisfied: parso>=0.3.0 in /venv/lib/python3.6/site-packages (from jedi->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.4.0)
    Requirement already satisfied: ipykernel in /venv/lib/python3.6/site-packages (from nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (5.1.0)
    Requirement already satisfied: coverage in /venv/lib/python3.6/site-packages (from nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (4.5.3)
    Requirement already satisfied: pytest>=2.8 in /venv/lib/python3.6/site-packages (from nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (4.4.0)
    Requirement already satisfied: nbformat in /venv/lib/python3.6/site-packages (from nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (4.4.0)
    Requirement already satisfied: jupyter-client in /venv/lib/python3.6/site-packages (from nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (5.2.4)
    Requirement already satisfied: pillow>=4.3.0 in /venv/lib/python3.6/site-packages (from scikit-image->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (6.0.0)
    Requirement already satisfied: networkx>=2.0 in /venv/lib/python3.6/site-packages (from scikit-image->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.3)
    Requirement already satisfied: matplotlib!=3.0.0,>=2.0.0 in /venv/lib/python3.6/site-packages (from scikit-image->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (3.0.3)
    Requirement already satisfied: imageio>=2.0.1 in /venv/lib/python3.6/site-packages (from scikit-image->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.5.0)
    Requirement already satisfied: PyWavelets>=0.4.0 in /venv/lib/python3.6/site-packages (from scikit-image->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.0.3)
    Requirement already satisfied: requests>=2.7.9 in /venv/lib/python3.6/site-packages (from codecov->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.21.0)
    Requirement already satisfied: pyyaml in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (5.1)
    Requirement already satisfied: cloudpickle>=0.2.2 in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.8.1)
    Requirement already satisfied: msgpack in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.6.1)
    Requirement already satisfied: tblib in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.3.2)
    Requirement already satisfied: toolz>=0.7.4 in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.9.0)
    Requirement already satisfied: zict>=0.1.3 in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.1.4)
    Requirement already satisfied: dask>=0.18.0 in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.2.0)
    Requirement already satisfied: tornado>=5 in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (6.0.2)
    Requirement already satisfied: sortedcontainers!=2.0.0,!=2.0.1 in /venv/lib/python3.6/site-packages (from distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.1.0)
    Requirement already satisfied: traitlets>=4.1.0 in /venv/lib/python3.6/site-packages (from ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (4.3.2)
    Requirement already satisfied: ipython>=5.0.0 in /venv/lib/python3.6/site-packages (from ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (7.4.0)
    Requirement already satisfied: attrs>=17.4.0 in /venv/lib/python3.6/site-packages (from pytest>=2.8->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (19.1.0)
    Requirement already satisfied: more-itertools>=4.0.0; python_version > "2.7" in /venv/lib/python3.6/site-packages (from pytest>=2.8->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (7.0.0)
    Requirement already satisfied: pluggy>=0.9 in /venv/lib/python3.6/site-packages (from pytest>=2.8->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.9.0)
    Requirement already satisfied: py>=1.5.0 in /venv/lib/python3.6/site-packages (from pytest>=2.8->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.8.0)
    Requirement already satisfied: setuptools in /venv/lib/python3.6/site-packages (from pytest>=2.8->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (40.6.2)
    Requirement already satisfied: atomicwrites>=1.0 in /venv/lib/python3.6/site-packages (from pytest>=2.8->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.3.0)
    Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /venv/lib/python3.6/site-packages (from nbformat->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (3.0.1)
    Requirement already satisfied: jupyter-core in /venv/lib/python3.6/site-packages (from nbformat->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (4.4.0)
    Requirement already satisfied: ipython-genutils in /venv/lib/python3.6/site-packages (from nbformat->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.2.0)
    Requirement already satisfied: pyzmq>=13 in /venv/lib/python3.6/site-packages (from jupyter-client->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (18.0.1)
    Requirement already satisfied: python-dateutil>=2.1 in /venv/lib/python3.6/site-packages (from jupyter-client->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.8.0)
    Requirement already satisfied: cycler>=0.10 in /venv/lib/python3.6/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.10.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /venv/lib/python3.6/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.4.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /venv/lib/python3.6/site-packages (from matplotlib!=3.0.0,>=2.0.0->scikit-image->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.0.1)
    Requirement already satisfied: idna<2.9,>=2.5 in /venv/lib/python3.6/site-packages (from requests>=2.7.9->codecov->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.8)
    Requirement already satisfied: certifi>=2017.4.17 in /venv/lib/python3.6/site-packages (from requests>=2.7.9->codecov->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2019.3.9)
    Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /venv/lib/python3.6/site-packages (from requests>=2.7.9->codecov->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (3.0.4)
    Requirement already satisfied: urllib3<1.25,>=1.21.1 in /venv/lib/python3.6/site-packages (from requests>=2.7.9->codecov->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.24.1)
    Requirement already satisfied: heapdict in /venv/lib/python3.6/site-packages (from zict>=0.1.3->distributed>=1.21.8->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (1.0.0)
    Requirement already satisfied: backcall in /venv/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.1.0)
    Requirement already satisfied: pygments in /venv/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.3.1)
    Requirement already satisfied: pexpect; sys_platform != "win32" in /venv/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (4.7.0)
    Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /venv/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (2.0.9)
    Requirement already satisfied: pickleshare in /venv/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.7.5)
    Requirement already satisfied: pyrsistent>=0.14.0 in /venv/lib/python3.6/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.14.11)
    Requirement already satisfied: ptyprocess>=0.5 in /venv/lib/python3.6/site-packages (from pexpect; sys_platform != "win32"->ipython>=5.0.0->ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.6.0)
    Requirement already satisfied: wcwidth in /venv/lib/python3.6/site-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython>=5.0.0->ipykernel->nbval->devito==3.4+873.gc57eec36->-r requirements.txt (line 3)) (0.1.7)
    Installing collected packages: devito
      Running setup.py install for devito ... done
    Successfully installed devito-3.4+873.gc57eec36

Once the above setup has finished, we should run a quick sanity check that we have everything that we need. The following cell should simply complete without any errors.

```python
from devito import *
from examples.seismic import Model
```

For one section of this tutorial we will have a quick look at the stencil compiler YASK. To verify YASK works smoothly you can run:

```python
import yask
```

## Instructors and helpers

This tutorial will be given by:

* **Navjot Kukreja**
* **Lucas Cavalcante**
TODO

## Learning objectives

* How to use [SymPy](http://www.sympy.org) and [Devito](http://www.devitoproject.org) to create simple finite difference expressions from governing equations
* Creating Devito operators to perform highly optimized stencil computations from the symbolic kernel definitions
* Create basic seismic modelling operator to model wave propagation for a seismic survey
* Implement a functional FWI algorithm usign high-level components from [Devito](http://www.devitoproject.org) and `scipy.optimize`
* Gain an overview of the various performance optimization techniques used in Devito operators

TODO

## Outline

TODO

* [Session 1: **Introduction to Devito**](01_introduction.ipynb)
  * Functions and derivatives **[5min]**
  * Exercise: A linear convection operator **[10min]**
  * Second derivatives and high-order stencils **[5min]**
  * Exercise 2: Making a wave! **[10min]**

* Session 2: **Seismic Imaging**
    * [Full Waveform Inversion (FWI) with Devito](02a_fwi.ipynb)
    * [Integration with Scipy.optimize](02b_scipy_optimize.ipynb)
    * [Distributed processing with Dask](02c_dask.ipynb)
    * [Advanced imaging with Skimage](02d_skimage_tv.ipynb)

* Session 3: [**Performance Optimization and Analysis**](03_performance.ipynb)
    * Introduction to performance optimization in Devito **[2min]**
    * Setup for shared-memory parallelism **[5min]**
    * Devito Symbolic Engine (DSE) **[5min]**
    * Devito Loop Engine (DLE) **[5min]**
    * Exercise 4: performance analysis of a TTI forward operator **[8min]**
    * A sneak peek at the YASK backend **[5min]**

### Bonus Material and further reading

TODO

* [Opesci project webpage](http://www.opesci.org/)
  * [Devito documentation](http://www.opesci.org/devito/)
* More detailed [introductory tutorials](http://www.opesci.org/devito/tutorials.html), covering the following topics:
  * Introduction to Devito with CFD
  * Introdcution to seismic imaging

### References

* M. Lange, N. Kukreja, F. Luporini, M. Louboutin, C. Yount, J. Hückelheim and G. Gorman. Optimised finite difference computation from symbolic equations. Accepted for publication in Proceedings of the 15th Python in Science Conference, 2017. [[doi:10.25080/shinma-7f4c6e7-00d](http://conference.scipy.org/proceedings/scipy2017/michael_lange.html)] [[arxiv](http://arxiv.org/abs/1707.03776)]

* M. Louboutin, M. Lange, N. Kukreja, F. Herrmann, and G. Gorman. _Performance
prediction of finite-difference solvers for different computer architectures_. Accepted
for publication in Computers & Geosciences, 2016, [doi:10.1016/j.cageo.2017.04.014](http://www.sciencedirect.com/science/article/pii/S0098300416304034)

This notebook is part of the tutorial "Optimised Symbolic Finite Difference Computation with Devito" presented at University of Sao Paulo April 2019.

```python

```


```python
#mapping some polygons -
import pandas as pd
import geopandas as gpd
from shapely.geometry.polygon import Polygon
import cartopy.crs as ccrs
import cartopy.feature as cfeature
import matplotlib.pyplot as plt
from matplotlib import cm
import folium
import numpy as np
%matplotlib inline
```

```python
#dataframes for csv files - Buffers
dfbuffer = pd.read_csv('bufferlatlongV2.csv', delimiter = ',').astype(float)
dfhurdat = pd.read_csv('hurdatlatlong.csv', delimiter = ',')
dfcoast = pd.read_csv('coastlatlongV2.csv', delimiter = ',').astype(float)
dfbasin = pd.read_csv('jamesbasinlatlongV2.csv', delimiter = ',').astype(float)

```

```python
#create the polygons
buffer_geom =Polygon(zip(dfbuffer['Lon'],dfbuffer['Lat']))
basin_geom = Polygon(zip(dfbasin['Lon'],dfbasin['Lat']))
crs = {'init' : 'epsg:4326'}
bufferpoly = gpd.GeoDataFrame(index = [0], crs = crs, geometry = [buffer_geom])
bufferpoly.to_file(filename = 'buffer.geojson', driver = 'GeoJSON')
basinpoly = gpd.GeoDataFrame(index = [0], crs = crs, geometry = [basin_geom])
basinpoly.to_file(filename = 'basin.geojson', driver = 'GeoJSON')
```

    /Users/williampc/opt/anaconda3/envs/geop/lib/python3.9/site-packages/pyproj/crs/crs.py:53: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
      return _prepare_from_string(" ".join(pjargs))

```python
import pandas as pd
from datetime import datetime
def lat_lon_to_float (v):
    """Convert strings from NHC to float locations"""
    if (v[-1] == 'S') or (v[-1] == 'W'):
        multiplier = -1
    else:
        multiplier = 1
    return float(v[:-1])*multiplier
```

```python
hurdata = []
with open ('hurdat2.txt', 'r') as f:
    for line in f.readlines():
        if line.startswith('AL'):
            storm_id = line.split(',')
            storm_number = storm_id[0].strip()
            storm_name = storm_id[1].strip()
        else:
            location_line = line.split(',')
            dt = datetime.strptime(location_line[0] + location_line[1],"%Y%m%d %H%M")
            storm_status = location_line[3].strip()
            storm_lat = lat_lon_to_float(location_line[4].strip())
            storm_lon = lat_lon_to_float(location_line[5].strip())
            max_speed = float(location_line[6].strip())
            hurdata.append([storm_number,storm_name,storm_status,storm_lat,storm_lon,dt,max_speed])
```

```python
df = pd.DataFrame(hurdata, columns = ['Storm Number','Storm Name', 'Storm Status', 'Lat', 'Lon','Time', 'Max Speed'])
df.head()
len(df)

```

    51817

```python
from sympy import Point, Polygon
import pandas as pd
import geopandas as gpd
from shapely.geometry import *
# # #changing to a GeoDataFrame to create geometry series
# hurdatgdf = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(df.Lon,df.Lat))
# # hurdatgdf.head()
# # len(hurdatgdf)
# T = pd.merge(dfbuffer, dfhurdat, how='inner', on=['Lat', 'Lon'])
# print(T)
# len(T)
# T
dfinbuffer = pd.merge(dfhurdat,dfbuffer,on=['Lat','Lon'])
pd.set_option("display.max_rows", None, "display.max_columns", None)
print(dfinbuffer)
```

        Unnamed: 0 Storm Number Storm Name Storm Status   Lat   Lon  \
    0         2874     AL021876    UNNAMED           HU  36.0 -77.3
    1         3620     AL021879    UNNAMED           HU  37.3 -75.4
    2         3620     AL021879    UNNAMED           HU  37.3 -75.4
    3         4906     AL081885    UNNAMED           EX  39.0 -78.0
    4         4906     AL081885    UNNAMED           EX  39.0 -78.0
    5         4906     AL081885    UNNAMED           EX  39.0 -78.0
    6         4906     AL081885    UNNAMED           EX  39.0 -78.0
    7         4906     AL081885    UNNAMED           EX  39.0 -78.0
    8         4906     AL081885    UNNAMED           EX  39.0 -78.0
    9         4947     AL021886    UNNAMED           TD  38.4 -76.9
    10        4947     AL021886    UNNAMED           TD  38.4 -76.9
    11        4947     AL021886    UNNAMED           TD  38.4 -76.9
    12        4947     AL021886    UNNAMED           TD  38.4 -76.9
    13        4947     AL021886    UNNAMED           TD  38.4 -76.9
    14        4947     AL021886    UNNAMED           TD  38.4 -76.9
    15        8951     AL031899    UNNAMED           HU  36.4 -75.3
    16        8951     AL031899    UNNAMED           HU  36.4 -75.3
    17        8951     AL031899    UNNAMED           HU  36.4 -75.3
    18        8951     AL031899    UNNAMED           HU  36.4 -75.3
    19        9621     AL031901    UNNAMED           HU  36.1 -75.6
    20        9621     AL031901    UNNAMED           HU  36.1 -75.6
    21        9621     AL031901    UNNAMED           HU  36.1 -75.6
    22        9621     AL031901    UNNAMED           HU  36.1 -75.6
    23        9621     AL031901    UNNAMED           HU  36.1 -75.6
    24        9621     AL031901    UNNAMED           HU  36.1 -75.6
    25        9621     AL031901    UNNAMED           HU  36.1 -75.6
    26        9622     AL031901    UNNAMED           HU  36.0 -75.8
    27        9622     AL031901    UNNAMED           HU  36.0 -75.8
    28        9622     AL031901    UNNAMED           HU  36.0 -75.8
    29       45637     AL072007  GABRIELLE           TS  36.0 -75.8
    30       45637     AL072007  GABRIELLE           TS  36.0 -75.8
    31       45637     AL072007  GABRIELLE           TS  36.0 -75.8
    32        9901     AL101901    UNNAMED           EX  37.8 -81.4
    33        9901     AL101901    UNNAMED           EX  37.8 -81.4
    34       20141     AL071943    UNNAMED           TS  37.7 -75.7
    35       20141     AL071943    UNNAMED           TS  37.7 -75.7
    36       20141     AL071943    UNNAMED           TS  37.7 -75.7
    37       20141     AL071943    UNNAMED           TS  37.7 -75.7
    38       20141     AL071943    UNNAMED           TS  37.7 -75.7
    39       20141     AL071943    UNNAMED           TS  37.7 -75.7
    40       22903     AL021952       ABLE           TS  36.6 -79.9
    41       22903     AL021952       ABLE           TS  36.6 -79.9
    42       22903     AL021952       ABLE           TS  36.6 -79.9
    43       22903     AL021952       ABLE           TS  36.6 -79.9
    44       22903     AL021952       ABLE           TS  36.6 -79.9
    45       22903     AL021952       ABLE           TS  36.6 -79.9
    46       27462     AL131964    UNNAMED           TD  36.2 -75.5
    47       27462     AL131964    UNNAMED           TD  36.2 -75.5
    48       27462     AL131964    UNNAMED           TD  36.2 -75.5
    49       27462     AL131964    UNNAMED           TD  36.2 -75.5
    50       27462     AL131964    UNNAMED           TD  36.2 -75.5
    51       28542     AL141967      DORIA           TS  37.3 -75.3
    52       28542     AL141967      DORIA           TS  37.3 -75.3
    53       29875     AL011970       ALMA           EX  37.0 -75.5
    54       32268     AL131975     ELOISE           EX  37.5 -81.5
    55       32268     AL131975     ELOISE           EX  37.5 -81.5
    56       38896     AL031994      BERYL           TD  37.5 -81.5
    57       38896     AL031994      BERYL           TD  37.5 -81.5
    58       40093     AL061996       FRAN           TD  39.2 -79.9
    59       40093     AL061996       FRAN           TD  39.2 -79.9
    60       42103     AL012001    ALLISON           SD  35.9 -76.8
    61       42103     AL012001    ALLISON           SD  35.9 -76.8

                       Time
    0   1876-09-17 18:00:00
    1   1879-08-18 18:00:00
    2   1879-08-18 18:00:00
    3   1885-10-13 18:00:00
    4   1885-10-13 18:00:00
    5   1885-10-13 18:00:00
    6   1885-10-13 18:00:00
    7   1885-10-13 18:00:00
    8   1885-10-13 18:00:00
    9   1886-06-23 06:00:00
    10  1886-06-23 06:00:00
    11  1886-06-23 06:00:00
    12  1886-06-23 06:00:00
    13  1886-06-23 06:00:00
    14  1886-06-23 06:00:00
    15  1899-08-19 00:00:00
    16  1899-08-19 00:00:00
    17  1899-08-19 00:00:00
    18  1899-08-19 00:00:00
    19  1901-07-11 06:00:00
    20  1901-07-11 06:00:00
    21  1901-07-11 06:00:00
    22  1901-07-11 06:00:00
    23  1901-07-11 06:00:00
    24  1901-07-11 06:00:00
    25  1901-07-11 06:00:00
    26  1901-07-11 07:00:00
    27  1901-07-11 07:00:00
    28  1901-07-11 07:00:00
    29  2007-09-10 00:00:00
    30  2007-09-10 00:00:00
    31  2007-09-10 00:00:00
    32  1901-09-29 00:00:00
    33  1901-09-29 00:00:00
    34  1943-10-01 00:00:00
    35  1943-10-01 00:00:00
    36  1943-10-01 00:00:00
    37  1943-10-01 00:00:00
    38  1943-10-01 00:00:00
    39  1943-10-01 00:00:00
    40  1952-09-01 00:00:00
    41  1952-09-01 00:00:00
    42  1952-09-01 00:00:00
    43  1952-09-01 00:00:00
    44  1952-09-01 00:00:00
    45  1952-09-01 00:00:00
    46  1964-07-24 06:00:00
    47  1964-07-24 06:00:00
    48  1964-07-24 06:00:00
    49  1964-07-24 06:00:00
    50  1964-07-24 06:00:00
    51  1967-09-16 18:00:00
    52  1967-09-16 18:00:00
    53  1970-05-27 06:00:00
    54  1975-09-24 18:00:00
    55  1975-09-24 18:00:00
    56  1994-08-17 18:00:00
    57  1994-08-17 18:00:00
    58  1996-09-07 00:00:00
    59  1996-09-07 00:00:00
    60  2001-06-15 18:00:00
    61  2001-06-15 18:00:00

```python
# total map with all storms and buffers
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature
```

```python
plot_crs = ccrs.LambertConformal(central_longitude =-100., central_latitude = 45)
data_crs = ccrs.PlateCarree()
```

```python
import matplotlib.patches as mpatches
import matplotlib.pyplot as plt
import shapely.geometry as sgeom
import cartopy.crs as ccrs
import cartopy.io.shapereader as shpreader

def basincoords():
    """
    Return a list of latitudes and a list of longitudes (lons, lats)
    for James River Basin

    """
    basinlon = dfbasin['Lon']
    basinlat = dfbasin['Lat']

    return basinlon, basinlat

def buffercoords():
    """
    Return a list of latitudes and a list of longitudes (lons, lats)
    for James River Basin

    """
    bufferlon = dfbuffer['Lon']
    bufferlat = dfbuffer['Lat']

    return bufferlon, bufferlat
bufferlon, bufferlat = buffercoords()
basinlon, basinlat = basincoords()

#ax.set_title('James River Basin and Buffer - Virginia, USA')

# turn the lons and lats into a shapely LineString
buffer = sgeom.LineString(zip(bufferlon, bufferlat))
basin = sgeom.LineString(zip(basinlon, basinlat))

fig = plt.figure(figsize = (7,7))
ax = plt.subplot(1,1,1,projection = plot_crs)

ax.set_extent([-85,-70,32,40],data_crs)
ax.coastlines('50m', edgecolor = 'k', linewidth = 0.75)
ax.add_feature(cfeature.STATES, linewidth = 0.5)
ax.add_feature(cfeature.RIVERS, linewidth = 0.85)
ax.add_feature(cfeature.OCEAN)
ax.add_geometries([buffer], ccrs.PlateCarree(),facecolor='#C8A2C8', alpha=0.5)
ax.add_geometries([basin], ccrs.PlateCarree(),facecolor='rgb', edgecolor='k')

#for storm_number in vahurc['Storm Number'].unique():
    #data = df[jameshurc['Storm Number'] == storm_number]
    #print(vahurc)
ax.plot(dfinbuffer['Lon'], dfinbuffer['Lat'])#,dfbuffer['Lon'],dfbuffer['Lat'],dfbasin['Lat'],dfbasin['Lon'], transform = data_crs)
```

```python

```

```python

```


<!-- dom:TITLE: Week 9 March 1-5: Resampling Techniques, Bootstrap and Blocking -->
# Week 9 March 1-5: Resampling Techniques, Bootstrap and Blocking
<!-- dom:AUTHOR: Morten Hjorth-Jensen  Email morten.hjorth-jensen@fys.uio.no at Department of Physics and Center fo Computing in Science Education, University of Oslo, Oslo, Norway & Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, East Lansing, Michigan, USA -->
<!-- Author: -->
**Morten Hjorth-Jensen  Email morten.hjorth-jensen@fys.uio.no**, Department of Physics and Center fo Computing in Science Education, University of Oslo, Oslo, Norway and Department of Physics and Astronomy and Facility for Rare Ion Beams, Michigan State University, East Lansing, Michigan, USA

Date: **Mar 19, 2021**

Copyright 1999-2021, Morten Hjorth-Jensen  Email morten.hjorth-jensen@fys.uio.no. Released under CC Attribution-NonCommercial 4.0 license

## Overview of week 9, March 1-5
**Topics.**

* Top down approach first, what we need to code

* Resampling Techniques and statistics: Bootstrap and Blocking

**Teaching Material, videos and written material.**

* Overview video on the [Bootstrap method](https://www.youtube.com/watch?v=O_Fj4q8lgmc&ab_channel=MarinStatsLectures-RProgramming%26Statistics)

* These Lecture notes

* [Marius Johnson's Master thesis on the Blocking Method](https://www.duo.uio.no/bitstream/handle/10852/68360/PhysRevE.98.043304.pdf?sequence=2&isAllowed=y)

## The top-down approach, part 1

Last week we discusse dhow to implement a gradient descent method like the simplest possible gradient descent with a simple learning rate as parameter to tune. We repeat the codes here.

```python
%matplotlib inline

# 2-electron VMC code for 2dim quantum dot with importance sampling
# Using gaussian rng for new positions and Metropolis- Hastings
# Added energy minimization
# Common imports
from math import exp, sqrt
from random import random, seed, normalvariate
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import sys

# Trial wave function for the 2-electron quantum dot in two dims
def WaveFunction(r,alpha,beta):
    r1 = r[0,0]**2 + r[0,1]**2
    r2 = r[1,0]**2 + r[1,1]**2
    r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2)
    deno = r12/(1+beta*r12)
    return exp(-0.5*alpha*(r1+r2)+deno)

# Local energy  for the 2-electron quantum dot in two dims, using analytical local energy
def LocalEnergy(r,alpha,beta):

    r1 = (r[0,0]**2 + r[0,1]**2)
    r2 = (r[1,0]**2 + r[1,1]**2)
    r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2)
    deno = 1.0/(1+beta*r12)
    deno2 = deno*deno
    return 0.5*(1-alpha*alpha)*(r1 + r2) +2.0*alpha + 1.0/r12+deno2*(alpha*r12-deno2+2*beta*deno-1.0/r12)

# Derivate of wave function ansatz as function of variational parameters
def DerivativeWFansatz(r,alpha,beta):

    WfDer  = np.zeros((2), np.double)
    r1 = (r[0,0]**2 + r[0,1]**2)
    r2 = (r[1,0]**2 + r[1,1]**2)
    r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2)
    deno = 1.0/(1+beta*r12)
    deno2 = deno*deno
    WfDer[0] = -0.5*(r1+r2)
    WfDer[1] = -r12*r12*deno2
    return  WfDer

# Setting up the quantum force for the two-electron quantum dot, recall that it is a vector
def QuantumForce(r,alpha,beta):

    qforce = np.zeros((NumberParticles,Dimension), np.double)
    r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2)
    deno = 1.0/(1+beta*r12)
    qforce[0,:] = -2*r[0,:]*alpha*(r[0,:]-r[1,:])*deno*deno/r12
    qforce[1,:] = -2*r[1,:]*alpha*(r[1,:]-r[0,:])*deno*deno/r12
    return qforce

# Computing the derivative of the energy and the energy
def EnergyMinimization(alpha, beta):

    NumberMCcycles= 10000
    # Parameters in the Fokker-Planck simulation of the quantum force
    D = 0.5
    TimeStep = 0.05
    # positions
    PositionOld = np.zeros((NumberParticles,Dimension), np.double)
    PositionNew = np.zeros((NumberParticles,Dimension), np.double)
    # Quantum force
    QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double)
    QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double)

    # seed for rng generator
    seed()
    energy = 0.0
    DeltaE = 0.0
    EnergyDer = np.zeros((2), np.double)
    DeltaPsi = np.zeros((2), np.double)
    DerivativePsiE = np.zeros((2), np.double)
    #Initial position
    for i in range(NumberParticles):
        for j in range(Dimension):
            PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep)
    wfold = WaveFunction(PositionOld,alpha,beta)
    QuantumForceOld = QuantumForce(PositionOld,alpha, beta)

    #Loop over MC MCcycles
    for MCcycle in range(NumberMCcycles):
        #Trial position moving one particle at the time
        for i in range(NumberParticles):
            for j in range(Dimension):
                PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\
                                       QuantumForceOld[i,j]*TimeStep*D
            wfnew = WaveFunction(PositionNew,alpha,beta)
            QuantumForceNew = QuantumForce(PositionNew,alpha, beta)
            GreensFunction = 0.0
            for j in range(Dimension):
                GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\
	                              (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\
                                      PositionNew[i,j]+PositionOld[i,j])

            GreensFunction = exp(GreensFunction)
            ProbabilityRatio = GreensFunction*wfnew**2/wfold**2
            #Metropolis-Hastings test to see whether we accept the move
            if random() <= ProbabilityRatio:
                for j in range(Dimension):
                    PositionOld[i,j] = PositionNew[i,j]
                    QuantumForceOld[i,j] = QuantumForceNew[i,j]
                wfold = wfnew
        DeltaE = LocalEnergy(PositionOld,alpha,beta)
        DerPsi = DerivativeWFansatz(PositionOld,alpha,beta)
        DeltaPsi += DerPsi
        energy += DeltaE
        DerivativePsiE += DerPsi*DeltaE

    # We calculate mean values
    energy /= NumberMCcycles
    DerivativePsiE /= NumberMCcycles
    DeltaPsi /= NumberMCcycles
    EnergyDer  = 2*(DerivativePsiE-DeltaPsi*energy)
    return energy, EnergyDer

#Here starts the main program with variable declarations
NumberParticles = 2
Dimension = 2
# guess for variational parameters
alpha = 0.95
beta = 0.3
# Set up iteration using stochastic gradient method
Energy = 0
EDerivative = np.zeros((2), np.double)
# Learning rate eta, max iterations, need to change to adaptive learning rate
eta = 0.01
MaxIterations = 50
iter = 0

Energies = np.zeros(MaxIterations)
EnergyDerivatives1 = np.zeros(MaxIterations)
EnergyDerivatives2 = np.zeros(MaxIterations)
AlphaValues = np.zeros(MaxIterations)
BetaValues = np.zeros(MaxIterations)

while iter < MaxIterations:
    Energy, EDerivative = EnergyMinimization(alpha,beta)
    alphagradient = EDerivative[0]
    betagradient = EDerivative[1]
    alpha -= eta*alphagradient
    beta -= eta*betagradient
    Energies[iter] = Energy
    EnergyDerivatives1[iter] = EDerivative[0]
    EnergyDerivatives2[iter] = EDerivative[1]
    AlphaValues[iter] = alpha
    BetaValues[iter] = beta
    iter += 1

#nice printout with Pandas
import pandas as pd
from pandas import DataFrame
pd.set_option('max_columns', 6)
data ={'Alpha':AlphaValues,'Beta':BetaValues,'Energy':Energies,'Alpha Derivative':EnergyDerivatives1,'Beta Derivative':EnergyDerivatives2}

frame = pd.DataFrame(data)
print(frame)
```

           Alpha      Beta    Energy  Alpha Derivative  Beta Derivative
    0   0.952471  0.302037  3.016387         -0.247135        -0.203671
    1   0.954795  0.303940  3.029544         -0.232340        -0.190346
    2   0.956858  0.305676  3.015198         -0.206349        -0.173568
    3   0.958977  0.307486  3.006236         -0.211873        -0.181037
    4   0.960851  0.309150  3.012114         -0.187402        -0.166353
    5   0.962882  0.310941  3.007021         -0.203075        -0.179168
    6   0.964631  0.312468  2.999393         -0.174974        -0.152620
    7   0.966714  0.314139  3.009335         -0.208262        -0.167151
    8   0.968463  0.315547  3.020319         -0.174855        -0.140742
    9   0.970040  0.316911  3.014918         -0.157758        -0.136428
    10  0.971736  0.318407  3.001852         -0.169535        -0.149638
    11  0.973292  0.319758  3.004794         -0.155687        -0.135037
    12  0.974575  0.320990  2.997130         -0.128237        -0.123228
    13  0.976202  0.322293  2.999610         -0.162677        -0.130268
    14  0.977609  0.323469  3.013079         -0.140761        -0.117682
    15  0.978797  0.324532  3.008125         -0.118754        -0.106286
    16  0.980063  0.325636  3.000419         -0.126608        -0.110420
    17  0.981101  0.326593  3.009269         -0.103811        -0.095629
    18  0.982087  0.327532  3.004387         -0.098621        -0.093912
    19  0.983044  0.328518  2.999726         -0.095676        -0.098639
    20  0.984256  0.329586  2.995441         -0.121211        -0.106785
    21  0.985339  0.330563  2.998627         -0.108343        -0.097689
    22  0.986245  0.331438  3.001941         -0.090580        -0.087516
    23  0.987186  0.332305  2.998803         -0.094048        -0.086666
    24  0.988028  0.333181  2.994558         -0.084252        -0.087604
    25  0.988790  0.333976  3.003508         -0.076148        -0.079508
    26  0.989604  0.334765  2.998644         -0.081400        -0.078893
    27  0.990332  0.335491  3.003694         -0.072851        -0.072578
    28  0.991020  0.336210  3.001020         -0.068826        -0.071905
    29  0.991715  0.336880  3.004798         -0.069469        -0.067018
    30  0.992301  0.337509  3.007172         -0.058630        -0.062906
    31  0.992932  0.338159  3.003623         -0.063095        -0.065009
    32  0.993519  0.338815  3.003110         -0.058667        -0.065637
    33  0.994145  0.339486  2.998508         -0.062584        -0.067014
    34  0.994722  0.340136  2.998125         -0.057742        -0.065075
    35  0.995276  0.340754  3.000603         -0.055382        -0.061730
    36  0.995745  0.341244  3.010393         -0.046852        -0.049045
    37  0.996204  0.341782  3.002136         -0.045980        -0.053780
    38  0.996676  0.342352  2.997916         -0.047181        -0.056985
    39  0.997130  0.342935  2.998770         -0.045353        -0.058314
    40  0.997575  0.343455  3.000545         -0.044469        -0.052040
    41  0.997916  0.343902  3.005916         -0.034128        -0.044679
    42  0.998283  0.344348  3.006643         -0.036676        -0.044615
    43  0.998697  0.344871  3.000212         -0.041480        -0.052282
    44  0.998969  0.345272  3.003977         -0.027170        -0.040150
    45  0.999345  0.345747  3.000847         -0.037629        -0.047456
    46  0.999667  0.346191  3.003118         -0.032147        -0.044381
    47  1.000018  0.346676  2.997754         -0.035130        -0.048558
    48  1.000334  0.347132  2.998584         -0.031554        -0.045563
    49  1.000564  0.347502  3.003228         -0.023081        -0.036966

## What have we done?
The exact energy is $3.0$ for an oscillator frequency $\omega =1$
(with $\hbar =1$). We note however that with this learning rate and
number of iterations, the energies and the derivatives are not yet
converged.

We can improve upon this by using the algorithms provided by the **optimize** package in Python.
One of these algorithms is  Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm.

The optimization problem is to minimize $f(\mathbf {x} )$ where
$\mathbf {x}$ is a vector in $R^{n}$, and $f$ is a differentiable
scalar function. There are no constraints on the values that $\mathbf{x}$ can take.

The algorithm begins at an initial estimate for the optimal value
$\mathbf {x}_{0}$ and proceeds iteratively to get a better estimate at
each stage.

The search direction $p_k$ at stage $k$ is given by the solution of the analogue of the Newton equation

$$
B_{k}\mathbf {p} _{k}=-\nabla f(\mathbf {x}_{k}),
$$

where $B_{k}$ is an approximation to the Hessian matrix, which is
updated iteratively at each stage, and $\nabla f(\mathbf {x} _{k})$
is the gradient of the function
evaluated at $x_k$.
A line search in the direction $p_k$ is then used to
find the next point $x_{k+1}$ by minimising

$$
f(\mathbf {x}_{k}+\alpha \mathbf {p}_{k}),
$$

over the scalar $\alpha > 0$.

## Code part 2
The modified code here uses the BFGS algorithm but performs now a
production run and writes to file all average values of the
energy.

```python
# 2-electron VMC code for 2dim quantum dot with importance sampling
# Using gaussian rng for new positions and Metropolis- Hastings
# Added energy minimization
from math import exp, sqrt
from random import random, seed, normalvariate
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
from scipy.optimize import minimize
import sys
import os

# Where to save data files
PROJECT_ROOT_DIR = "Results"
DATA_ID = "Results/EnergyMin"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

outfile = open(data_path("Energies.dat"),'w')

# Trial wave function for the 2-electron quantum dot in two dims
def WaveFunction(r,alpha,beta):
    r1 = r[0,0]**2 + r[0,1]**2
    r2 = r[1,0]**2 + r[1,1]**2
    r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2)
    deno = r12/(1+beta*r12)
    return exp(-0.5*alpha*(r1+r2)+deno)

# Local energy  for the 2-electron quantum dot in two dims, using analytical local energy
def LocalEnergy(r,alpha,beta):

    r1 = (r[0,0]**2 + r[0,1]**2)
    r2 = (r[1,0]**2 + r[1,1]**2)
    r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2)
    deno = 1.0/(1+beta*r12)
    deno2 = deno*deno
    return 0.5*(1-alpha*alpha)*(r1 + r2) +2.0*alpha + 1.0/r12+deno2*(alpha*r12-deno2+2*beta*deno-1.0/r12)

# Derivate of wave function ansatz as function of variational parameters
def DerivativeWFansatz(r,alpha,beta):

    WfDer  = np.zeros((2), np.double)
    r1 = (r[0,0]**2 + r[0,1]**2)
    r2 = (r[1,0]**2 + r[1,1]**2)
    r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2)
    deno = 1.0/(1+beta*r12)
    deno2 = deno*deno
    WfDer[0] = -0.5*(r1+r2)
    WfDer[1] = -r12*r12*deno2
    return  WfDer

# Setting up the quantum force for the two-electron quantum dot, recall that it is a vector
def QuantumForce(r,alpha,beta):

    qforce = np.zeros((NumberParticles,Dimension), np.double)
    r12 = sqrt((r[0,0]-r[1,0])**2 + (r[0,1]-r[1,1])**2)
    deno = 1.0/(1+beta*r12)
    qforce[0,:] = -2*r[0,:]*alpha*(r[0,:]-r[1,:])*deno*deno/r12
    qforce[1,:] = -2*r[1,:]*alpha*(r[1,:]-r[0,:])*deno*deno/r12
    return qforce

# Computing the derivative of the energy and the energy
def EnergyDerivative(x0):

    # Parameters in the Fokker-Planck simulation of the quantum force
    D = 0.5
    TimeStep = 0.05
    # positions
    PositionOld = np.zeros((NumberParticles,Dimension), np.double)
    PositionNew = np.zeros((NumberParticles,Dimension), np.double)
    # Quantum force
    QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double)
    QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double)

    energy = 0.0
    DeltaE = 0.0
    alpha = x0[0]
    beta = x0[1]
    EnergyDer = 0.0
    DeltaPsi = 0.0
    DerivativePsiE = 0.0
    #Initial position
    for i in range(NumberParticles):
        for j in range(Dimension):
            PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep)
    wfold = WaveFunction(PositionOld,alpha,beta)
    QuantumForceOld = QuantumForce(PositionOld,alpha, beta)

    #Loop over MC MCcycles
    for MCcycle in range(NumberMCcycles):
        #Trial position moving one particle at the time
        for i in range(NumberParticles):
            for j in range(Dimension):
                PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\
                                       QuantumForceOld[i,j]*TimeStep*D
            wfnew = WaveFunction(PositionNew,alpha,beta)
            QuantumForceNew = QuantumForce(PositionNew,alpha, beta)
            GreensFunction = 0.0
            for j in range(Dimension):
                GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\
	                              (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\
                                      PositionNew[i,j]+PositionOld[i,j])

            GreensFunction = exp(GreensFunction)
            ProbabilityRatio = GreensFunction*wfnew**2/wfold**2
            #Metropolis-Hastings test to see whether we accept the move
            if random() <= ProbabilityRatio:
                for j in range(Dimension):
                    PositionOld[i,j] = PositionNew[i,j]
                    QuantumForceOld[i,j] = QuantumForceNew[i,j]
                wfold = wfnew
        DeltaE = LocalEnergy(PositionOld,alpha,beta)
        DerPsi = DerivativeWFansatz(PositionOld,alpha,beta)
        DeltaPsi += DerPsi
        energy += DeltaE
        DerivativePsiE += DerPsi*DeltaE

    # We calculate mean values
    energy /= NumberMCcycles
    DerivativePsiE /= NumberMCcycles
    DeltaPsi /= NumberMCcycles
    EnergyDer  = 2*(DerivativePsiE-DeltaPsi*energy)
    return EnergyDer

# Computing the expectation value of the local energy
def Energy(x0):
    # Parameters in the Fokker-Planck simulation of the quantum force
    D = 0.5
    TimeStep = 0.05
    # positions
    PositionOld = np.zeros((NumberParticles,Dimension), np.double)
    PositionNew = np.zeros((NumberParticles,Dimension), np.double)
    # Quantum force
    QuantumForceOld = np.zeros((NumberParticles,Dimension), np.double)
    QuantumForceNew = np.zeros((NumberParticles,Dimension), np.double)

    energy = 0.0
    DeltaE = 0.0
    alpha = x0[0]
    beta = x0[1]
    #Initial position
    for i in range(NumberParticles):
        for j in range(Dimension):
            PositionOld[i,j] = normalvariate(0.0,1.0)*sqrt(TimeStep)
    wfold = WaveFunction(PositionOld,alpha,beta)
    QuantumForceOld = QuantumForce(PositionOld,alpha, beta)

    #Loop over MC MCcycles
    for MCcycle in range(NumberMCcycles):
        #Trial position moving one particle at the time
        for i in range(NumberParticles):
            for j in range(Dimension):
                PositionNew[i,j] = PositionOld[i,j]+normalvariate(0.0,1.0)*sqrt(TimeStep)+\
                                       QuantumForceOld[i,j]*TimeStep*D
            wfnew = WaveFunction(PositionNew,alpha,beta)
            QuantumForceNew = QuantumForce(PositionNew,alpha, beta)
            GreensFunction = 0.0
            for j in range(Dimension):
                GreensFunction += 0.5*(QuantumForceOld[i,j]+QuantumForceNew[i,j])*\
	                              (D*TimeStep*0.5*(QuantumForceOld[i,j]-QuantumForceNew[i,j])-\
                                      PositionNew[i,j]+PositionOld[i,j])

            GreensFunction = exp(GreensFunction)
            ProbabilityRatio = GreensFunction*wfnew**2/wfold**2
            #Metropolis-Hastings test to see whether we accept the move
            if random() <= ProbabilityRatio:
                for j in range(Dimension):
                    PositionOld[i,j] = PositionNew[i,j]
                    QuantumForceOld[i,j] = QuantumForceNew[i,j]
                wfold = wfnew
        DeltaE = LocalEnergy(PositionOld,alpha,beta)
        energy += DeltaE
        if Printout:
           outfile.write('%f\n' %(DeltaE))#  ('%f\n' %(energy/(MCcycle+1.0)))
    # We calculate mean values
    energy /= NumberMCcycles
    return energy

#Here starts the main program with variable declarations
NumberParticles = 2
Dimension = 2
# seed for rng generator
seed()
# Monte Carlo cycles for parameter optimization
Printout = False
NumberMCcycles= 10000
# guess for variational parameters
x0 = np.array([0.9,0.2])
# Using Broydens method to find optimal parameters
res = minimize(Energy, x0, method='BFGS', jac=EnergyDerivative, options={'gtol': 1e-4,'disp': True})
x0 = res.x
# Compute the energy again with the optimal parameters and increased number of Monte Cycles
NumberMCcycles= 2**19
Printout = True
FinalEnergy = Energy(x0)
EResult = np.array([FinalEnergy,FinalEnergy])
outfile.close()
#nice printout with Pandas
import pandas as pd
from pandas import DataFrame
data ={'Optimal Parameters':x0, 'Final Energy':EResult}
frame = pd.DataFrame(data)
print(frame)
```

    Warning: Desired error not necessarily achieved due to precision loss.
             Current function value: 2.998422
             Iterations: 3
             Function evaluations: 22
             Gradient evaluations: 10
       Optimal Parameters  Final Energy
    0            0.962938      3.000951
    1            0.457425      3.000951

Note that the **minimize** function returns the final values for the
variable $\alpha=x0[0]$ and $\beta=x0[1]$ in the array $x$.

When we have found the minimum, we use these optimal parameters to perform a production run of energies.
The output is in turn written to file and is used, together with resampling methods like the **blocking method**,
to obtain the best possible estimate for the standard deviation.   The optimal minimum is, even with our guess, rather close to the exact value of $3.0$ a.u.

The [sampling
functions](https://github.com/CompPhysics/ComputationalPhysics2/tree/gh-pages/doc/Programs/Resampling)
can be used to perform both a blocking analysis, or a standard
bootstrap and jackknife analysis.

## How do we proceed?

There are several paths which can be chosen. One is to extend the
brute force gradient descent method with an adapative stochastic
gradient. There are several examples of this. A recent approach based
on [the Langevin equations](https://arxiv.org/pdf/1805.09416.pdf)
seems like a promising approach for general and possibly non-convex
optimization problems.

Here we would like to point out that our next step is now to use the
optimal values for our variational parameters and use these as inputs
to a production run. Here we would output values of the energy and
perform for example a blocking analysis of the results in order to get
a best possible estimate of the standard deviation.

## Resampling analysis

The next step is then to use the above data sets and perform a
resampling analysis, either using say the Bootstrap method or the
Blocking method. Since the data will be correlated, we would recommend
to use the non-iid Bootstrap code here. The theoretical background for these resampling methods is found in the [statistical analysis lecture notes](http://compphysics.github.io/ComputationalPhysics2/doc/pub/statanalysis/html/statanalysis.html)

Here we have tailored the codes to the output file from the previous example. We present first the bootstrap resampling with non-iid stochastic event.

```python
# Common imports
import os

# Where to save the figures and data files
DATA_ID = "Results/EnergyMin"

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

infile = open(data_path("Energies.dat"),'r')

from numpy import std, mean, concatenate, arange, loadtxt, zeros, ceil
from numpy.random import randint
from time import time

def tsboot(data,statistic,R,l):
    t = zeros(R); n = len(data); k = int(ceil(float(n)/l));
    inds = arange(n); t0 = time()

    # time series bootstrap
    for i in range(R):
        # construct bootstrap sample from
        # k chunks of data. The chunksize is l
        _data = concatenate([data[j:j+l] for j in randint(0,n-l,k)])[0:n];
        t[i] = statistic(_data)

    # analysis
    print ("Runtime: %g sec" % (time()-t0)); print ("Bootstrap Statistics :")
    print ("original           bias      std. error")
    print ("%8g %14g %15g" % (statistic(data), \
                             mean(t) - statistic(data), \
                             std(t) ))
    return t
# Read in data
X = loadtxt(infile)
# statistic to be estimated. Takes two args.
# arg1: the data
def stat(data):
    return mean(data)
t = tsboot(X, stat, 2**12, 2**10)
```

    Runtime: 2.91977 sec
    Bootstrap Statistics :
    original           bias      std. error
     3.00095   -3.49515e-06     0.000311354

The blocking code, based on the article of [Marius Jonsson](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.98.043304) is given here

```python
# Common imports
import os

# Where to save the figures and data files
DATA_ID = "Results/EnergyMin"

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

infile = open(data_path("Energies.dat"),'r')

from numpy import log2, zeros, mean, var, sum, loadtxt, arange, array, cumsum, dot, transpose, diagonal, sqrt
from numpy.linalg import inv

def block(x):
    # preliminaries
    n = len(x)
    d = int(log2(n))
    s, gamma = zeros(d), zeros(d)
    mu = mean(x)

    # estimate the auto-covariance and variances
    # for each blocking transformation
    for i in arange(0,d):
        n = len(x)
        # estimate autocovariance of x
        gamma[i] = (n)**(-1)*sum( (x[0:(n-1)]-mu)*(x[1:n]-mu) )
        # estimate variance of x
        s[i] = var(x)
        # perform blocking transformation
        x = 0.5*(x[0::2] + x[1::2])

    # generate the test observator M_k from the theorem
    M = (cumsum( ((gamma/s)**2*2**arange(1,d+1)[::-1])[::-1] )  )[::-1]

    # we need a list of magic numbers
    q =array([6.634897,9.210340, 11.344867, 13.276704, 15.086272, 16.811894, 18.475307, 20.090235, 21.665994, 23.209251, 24.724970, 26.216967, 27.688250, 29.141238, 30.577914, 31.999927, 33.408664, 34.805306, 36.190869, 37.566235, 38.932173, 40.289360, 41.638398, 42.979820, 44.314105, 45.641683, 46.962942, 48.278236, 49.587884, 50.892181])

    # use magic to determine when we should have stopped blocking
    for k in arange(0,d):
        if(M[k] < q[k]):
            break
    if (k >= d-1):
        print("Warning: Use more data")
    return mu, s[k]/2**(d-k)

x = loadtxt(infile)
(mean, var) = block(x)
std = sqrt(var)
import pandas as pd
from pandas import DataFrame
data ={'Mean':[mean], 'STDev':[std]}
frame = pd.DataFrame(data,index=['Values'])
print(frame)
```

                Mean     STDev
    Values  3.000951  0.000314

## Why resampling methods ?
**Statistical analysis.**

    * Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods

    * The results can be analysed with the same statistical tools as we would use analysing experimental data.

    * As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.

## Statistical analysis
    * As in other experiments, many numerical  experiments have two classes of errors:

      * Statistical errors

      * Systematical errors

    * Statistical errors can be estimated using standard tools from statistics

    * Systematical errors are method specific and must be treated differently from case to case.

## Statistics
The *probability distribution function (PDF)* is a function
$p(x)$ on the domain which, in the discrete case, gives us the
probability or relative frequency with which these values of $X$ occur:

$$
p(x) = \mathrm{prob}(X=x)
$$

In the continuous case, the PDF does not directly depict the
actual probability. Instead we define the probability for the
stochastic variable to assume any value on an infinitesimal interval
around $x$ to be $p(x)dx$. The continuous function $p(x)$ then gives us
the *density* of the probability rather than the probability
itself. The probability for a stochastic variable to assume any value
on a non-infinitesimal interval $[a,\,b]$ is then just the integral:

$$
\mathrm{prob}(a\leq X\leq b) = \int_a^b p(x)dx
$$

Qualitatively speaking, a stochastic variable represents the values of
numbers chosen as if by chance from some specified PDF so that the
selection of a large set of these numbers reproduces this PDF.

## Statistics, moments
A particularly useful class of special expectation values are the
*moments*. The $n$-th moment of the PDF $p$ is defined as
follows:

$$
\langle x^n\rangle \equiv \int\! x^n p(x)\,dx
$$

The zero-th moment $\langle 1\rangle$ is just the normalization condition of
$p$. The first moment, $\langle x\rangle$, is called the *mean* of $p$
and often denoted by the letter $\mu$:

$$
\langle x\rangle = \mu \equiv \int\! x p(x)\,dx
$$

## Statistics, central moments
A special version of the moments is the set of *central moments*,
the n-th central moment defined as:

$$
\langle (x-\langle x \rangle )^n\rangle \equiv \int\! (x-\langle x\rangle)^n p(x)\,dx
$$

The zero-th and first central moments are both trivial, equal $1$ and
$0$, respectively. But the second central moment, known as the
*variance* of $p$, is of particular interest. For the stochastic
variable $X$, the variance is denoted as $\sigma^2_X$ or $\mathrm{var}(X)$:

<!-- Equation labels as ordinary links -->
<div id="_auto1"></div>

$$
\begin{equation}
\sigma^2_X\ \ =\ \ \mathrm{var}(X)  =  \langle (x-\langle x\rangle)^2\rangle =
\int\! (x-\langle x\rangle)^2 p(x)\,dx
\label{_auto1} \tag{1}
\end{equation}
$$

<!-- Equation labels as ordinary links -->
<div id="_auto2"></div>

$$
\begin{equation}
 =  \int\! \left(x^2 - 2 x \langle x\rangle^{2} +
  \langle x\rangle^2\right)p(x)\,dx
\label{_auto2} \tag{2}
\end{equation}
$$

<!-- Equation labels as ordinary links -->
<div id="_auto3"></div>

$$
\begin{equation}
 =  \langle x^2\rangle - 2 \langle x\rangle\langle x\rangle + \langle x\rangle^2
\label{_auto3} \tag{3}
\end{equation}
$$

<!-- Equation labels as ordinary links -->
<div id="_auto4"></div>

$$
\begin{equation}
 =  \langle x^2\rangle - \langle x\rangle^2
\label{_auto4} \tag{4}
\end{equation}
$$

The square root of the variance, $\sigma =\sqrt{\langle (x-\langle x\rangle)^2\rangle}$ is called the *standard deviation* of $p$. It is clearly just the RMS (root-mean-square)
value of the deviation of the PDF from its mean value, interpreted
qualitatively as the *spread* of $p$ around its mean.

## Statistics, covariance
Another important quantity is the so called covariance, a variant of
the above defined variance. Consider again the set $\{X_i\}$ of $n$
stochastic variables (not necessarily uncorrelated) with the
multivariate PDF $P(x_1,\dots,x_n)$. The *covariance* of two
of the stochastic variables, $X_i$ and $X_j$, is defined as follows:

$$
\mathrm{cov}(X_i,\,X_j) \equiv \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle
\nonumber
$$

<!-- Equation labels as ordinary links -->
<div id="eq:def_covariance"></div>

$$
\begin{equation}
=
\int\!\cdots\!\int\!(x_i-\langle x_i \rangle)(x_j-\langle x_j \rangle)\,
P(x_1,\dots,x_n)\,dx_1\dots dx_n
\label{eq:def_covariance} \tag{5}
\end{equation}
$$

with

$$
\langle x_i\rangle =
\int\!\cdots\!\int\!x_i\,P(x_1,\dots,x_n)\,dx_1\dots dx_n
$$

## Statistics, more covariance
If we consider the above covariance as a matrix $C_{ij}=\mathrm{cov}(X_i,\,X_j)$, then the diagonal elements are just the familiar
variances, $C_{ii} = \mathrm{cov}(X_i,\,X_i) = \mathrm{var}(X_i)$. It turns out that
all the off-diagonal elements are zero if the stochastic variables are
uncorrelated. This is easy to show, keeping in mind the linearity of
the expectation value. Consider the stochastic variables $X_i$ and
$X_j$, ($i\neq j$):

<!-- Equation labels as ordinary links -->
<div id="_auto5"></div>

$$
\begin{equation}
\mathrm{cov}(X_i,\,X_j) = \langle(x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle
\label{_auto5} \tag{6}
\end{equation}
$$

<!-- Equation labels as ordinary links -->
<div id="_auto6"></div>

$$
\begin{equation}
=\langle x_i x_j - x_i\langle x_j\rangle - \langle x_i\rangle x_j + \langle x_i\rangle\langle x_j\rangle\rangle
\label{_auto6} \tag{7}
\end{equation}
$$

<!-- Equation labels as ordinary links -->
<div id="_auto7"></div>

$$
\begin{equation}
=\langle x_i x_j\rangle - \langle x_i\langle x_j\rangle\rangle - \langle \langle x_i\rangle x_j\rangle +
\langle \langle x_i\rangle\langle x_j\rangle\rangle
\label{_auto7} \tag{8}
\end{equation}
$$

<!-- Equation labels as ordinary links -->
<div id="_auto8"></div>

$$
\begin{equation}
=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle - \langle x_i\rangle\langle x_j\rangle +
\langle x_i\rangle\langle x_j\rangle
\label{_auto8} \tag{9}
\end{equation}
$$

<!-- Equation labels as ordinary links -->
<div id="_auto9"></div>

$$
\begin{equation}
=\langle x_i x_j\rangle - \langle x_i\rangle\langle x_j\rangle
\label{_auto9} \tag{10}
\end{equation}
$$

## Statistics, independent variables
If $X_i$ and $X_j$ are independent, we get
$\langle x_i x_j\rangle =\langle x_i\rangle\langle x_j\rangle$, resulting in $\mathrm{cov}(X_i, X_j) = 0\ \ (i\neq j)$.

Also useful for us is the covariance of linear combinations of
stochastic variables. Let $\{X_i\}$ and $\{Y_i\}$ be two sets of
stochastic variables. Let also $\{a_i\}$ and $\{b_i\}$ be two sets of
scalars. Consider the linear combination:

$$
U = \sum_i a_i X_i \qquad V = \sum_j b_j Y_j
$$

By the linearity of the expectation value

$$
\mathrm{cov}(U, V) = \sum_{i,j}a_i b_j \mathrm{cov}(X_i, Y_j)
$$

## Statistics, more variance
Now, since the variance is just $\mathrm{var}(X_i) = \mathrm{cov}(X_i, X_i)$, we get
the variance of the linear combination $U = \sum_i a_i X_i$:

<!-- Equation labels as ordinary links -->
<div id="eq:variance_linear_combination"></div>

$$
\begin{equation}
\mathrm{var}(U) = \sum_{i,j}a_i a_j \mathrm{cov}(X_i, X_j)
\label{eq:variance_linear_combination} \tag{11}
\end{equation}
$$

And in the special case when the stochastic variables are
uncorrelated, the off-diagonal elements of the covariance are as we
know zero, resulting in:

2
2

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\mathrm{var}(\sum_i a_i X_i) = \sum_i a_i^2 \mathrm{var}(X_i)
$$

which will become very useful in our study of the error in the mean
value of a set of measurements.

## Statistics and stochastic processes
A *stochastic process* is a process that produces sequentially a
chain of values:

$$
\{x_1, x_2,\dots\,x_k,\dots\}.
$$

We will call these
values our *measurements* and the entire set as our measured
*sample*.  The action of measuring all the elements of a sample
we will call a stochastic *experiment* since, operationally,
they are often associated with results of empirical observation of
some physical or mathematical phenomena; precisely an experiment. We
assume that these values are distributed according to some
PDF $p_X^{\phantom X}(x)$, where $X$ is just the formal symbol for the
stochastic variable whose PDF is $p_X^{\phantom X}(x)$. Instead of
trying to determine the full distribution $p$ we are often only
interested in finding the few lowest moments, like the mean
$\mu_X^{\phantom X}$ and the variance $\sigma_X^{\phantom X}$.

<!-- !split  -->
## Statistics and sample variables
In practical situations a sample is always of finite size. Let that
size be $n$. The expectation value of a sample, the *sample mean*, is then defined as follows:

$$
\bar{x}_n \equiv \frac{1}{n}\sum_{k=1}^n x_k
$$

The *sample variance* is:

$$
\mathrm{var}(x) \equiv \frac{1}{n}\sum_{k=1}^n (x_k - \bar{x}_n)^2
$$

its square root being the *standard deviation of the sample*. The
*sample covariance* is:

$$
\mathrm{cov}(x)\equiv\frac{1}{n}\sum_{kl}(x_k - \bar{x}_n)(x_l - \bar{x}_n)
$$

## Statistics, sample variance and covariance
Note that the sample variance is the sample covariance without the
cross terms. In a similar manner as the covariance in Eq. ([5](#eq:def_covariance)) is a measure of the correlation between
two stochastic variables, the above defined sample covariance is a
measure of the sequential correlation between succeeding measurements
of a sample.

These quantities, being known experimental values, differ
significantly from and must not be confused with the similarly named
quantities for stochastic variables, mean $\mu_X$, variance $\mathrm{var}(X)$
and covariance $\mathrm{cov}(X,Y)$.

## Statistics, law of large numbers
The law of large numbers
states that as the size of our sample grows to infinity, the sample
mean approaches the true mean $\mu_X^{\phantom X}$ of the chosen PDF:

$$
\lim_{n\to\infty}\bar{x}_n = \mu_X^{\phantom X}
$$

The sample mean $\bar{x}_n$ works therefore as an estimate of the true
mean $\mu_X^{\phantom X}$.

What we need to find out is how good an approximation $\bar{x}_n$ is to
$\mu_X^{\phantom X}$. In any stochastic measurement, an estimated
mean is of no use to us without a measure of its error. A quantity
that tells us how well we can reproduce it in another experiment. We
are therefore interested in the PDF of the sample mean itself. Its
standard deviation will be a measure of the spread of sample means,
and we will simply call it the *error* of the sample mean, or
just sample error, and denote it by $\mathrm{err}_X^{\phantom X}$. In
practice, we will only be able to produce an *estimate* of the
sample error since the exact value would require the knowledge of the
true PDFs behind, which we usually do not have.

## Statistics, more on sample error
Let us first take a look at what happens to the sample error as the
size of the sample grows. In a sample, each of the measurements $x_i$
can be associated with its own stochastic variable $X_i$. The
stochastic variable $\overline X_n$ for the sample mean $\bar{x}_n$ is
then just a linear combination, already familiar to us:

$$
\overline X_n = \frac{1}{n}\sum_{i=1}^n X_i
$$

All the coefficients are just equal $1/n$. The PDF of $\overline X_n$,
denoted by $p_{\overline X_n}(x)$ is the desired PDF of the sample
means.

## Statistics
The probability density of obtaining a sample mean $\bar x_n$
is the product of probabilities of obtaining arbitrary values $x_1,
x_2,\dots,x_n$ with the constraint that the mean of the set $\{x_i\}$
is $\bar x_n$:

$$
p_{\overline X_n}(x) = \int p_X^{\phantom X}(x_1)\cdots
\int p_X^{\phantom X}(x_n)\
\delta\!\left(x - \frac{x_1+x_2+\dots+x_n}{n}\right)dx_n \cdots dx_1
$$

And in particular we are interested in its variance $\mathrm{var}(\overline X_n)$.

## Statistics, central limit theorem
It is generally not possible to express $p_{\overline X_n}(x)$ in a
closed form given an arbitrary PDF $p_X^{\phantom X}$ and a number
$n$. But for the limit $n\to\infty$ it is possible to make an
approximation. The very important result is called *the central limit theorem*. It tells us that as $n$ goes to infinity,
$p_{\overline X_n}(x)$ approaches a Gaussian distribution whose mean
and variance equal the true mean and variance, $\mu_{X}^{\phantom X}$
and $\sigma_{X}^{2}$, respectively:

<!-- Equation labels as ordinary links -->
<div id="eq:central_limit_gaussian"></div>

$$
\begin{equation}
\lim_{n\to\infty} p_{\overline X_n}(x) =
\left(\frac{n}{2\pi\mathrm{var}(X)}\right)^{1/2}
e^{-\frac{n(x-\bar x_n)^2}{2\mathrm{var}(X)}}
\label{eq:central_limit_gaussian} \tag{12}
\end{equation}
$$

## Statistics, more technicalities
The desired variance
$\mathrm{var}(\overline X_n)$, i.e. the sample error squared
$\mathrm{err}_X^2$, is given by:

<!-- Equation labels as ordinary links -->
<div id="eq:error_exact"></div>

$$
\begin{equation}
\mathrm{err}_X^2 = \mathrm{var}(\overline X_n) = \frac{1}{n^2}
\sum_{ij} \mathrm{cov}(X_i, X_j)
\label{eq:error_exact} \tag{13}
\end{equation}
$$

We see now that in order to calculate the exact error of the sample
with the above expression, we would need the true means
$\mu_{X_i}^{\phantom X}$ of the stochastic variables $X_i$. To
calculate these requires that we know the true multivariate PDF of all
the $X_i$. But this PDF is unknown to us, we have only got the measurements of
one sample. The best we can do is to let the sample itself be an
estimate of the PDF of each of the $X_i$, estimating all properties of
$X_i$ through the measurements of the sample.

## Statistics
Our estimate of $\mu_{X_i}^{\phantom X}$ is then the sample mean $\bar x$
itself, in accordance with the the central limit theorem:

$$
\mu_{X_i}^{\phantom X} = \langle x_i\rangle \approx \frac{1}{n}\sum_{k=1}^n x_k = \bar x
$$

Using $\bar x$ in place of $\mu_{X_i}^{\phantom X}$ we can give an
*estimate* of the covariance in Eq. ([13](#eq:error_exact))

$$
\mathrm{cov}(X_i, X_j) = \langle (x_i-\langle x_i\rangle)(x_j-\langle x_j\rangle)\rangle
\approx\langle (x_i - \bar x)(x_j - \bar{x})\rangle,
$$

resulting in

$$
\frac{1}{n} \sum_{l}^n \left(\frac{1}{n}\sum_{k}^n (x_k -\bar x_n)(x_l - \bar x_n)\right)=\frac{1}{n}\frac{1}{n} \sum_{kl} (x_k -\bar x_n)(x_l - \bar x_n)=\frac{1}{n}\mathrm{cov}(x)
$$

## Statistics and sample variance
By the same procedure we can use the sample variance as an
estimate of the variance of any of the stochastic variables $X_i$

$$
\mathrm{var}(X_i)=\langle x_i - \langle x_i\rangle\rangle \approx \langle x_i - \bar x_n\rangle\nonumber,
$$

which is approximated as

<!-- Equation labels as ordinary links -->
<div id="eq:var_estimate_i_think"></div>

$$
\begin{equation}
\mathrm{var}(X_i)\approx \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)=\mathrm{var}(x)
\label{eq:var_estimate_i_think} \tag{14}
\end{equation}
$$

Now we can calculate an estimate of the error
$\mathrm{err}_X^{\phantom X}$ of the sample mean $\bar x_n$:

$$
\mathrm{err}_X^2
=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) \nonumber
$$

$$
\approx\frac{1}{n^2}\sum_{ij}\frac{1}{n}\mathrm{cov}(x) =\frac{1}{n^2}n^2\frac{1}{n}\mathrm{cov}(x)\nonumber
$$

<!-- Equation labels as ordinary links -->
<div id="eq:error_estimate"></div>

$$
\begin{equation}
=\frac{1}{n}\mathrm{cov}(x)
\label{eq:error_estimate} \tag{15}
\end{equation}
$$

which is nothing but the sample covariance divided by the number of
measurements in the sample.

## Statistics, uncorrelated results

In the special case that the measurements of the sample are
uncorrelated (equivalently the stochastic variables $X_i$ are
uncorrelated) we have that the off-diagonal elements of the covariance
are zero. This gives the following estimate of the sample error:

$$
\mathrm{err}_X^2=\frac{1}{n^2}\sum_{ij} \mathrm{cov}(X_i, X_j) =
\frac{1}{n^2} \sum_i \mathrm{var}(X_i),
$$

resulting in

<!-- Equation labels as ordinary links -->
<div id="eq:error_estimate_uncorrel"></div>

$$
\begin{equation}
\mathrm{err}_X^2\approx \frac{1}{n^2} \sum_i \mathrm{var}(x)= \frac{1}{n}\mathrm{var}(x)
\label{eq:error_estimate_uncorrel} \tag{16}
\end{equation}
$$

where in the second step we have used Eq. ([14](#eq:var_estimate_i_think)).
The error of the sample is then just its standard deviation divided by
the square root of the number of measurements the sample contains.
This is a very useful formula which is easy to compute. It acts as a
first approximation to the error, but in numerical experiments, we
cannot overlook the always present correlations.

## Statistics, computations
For computational purposes one usually splits up the estimate of
$\mathrm{err}_X^2$, given by Eq. ([15](#eq:error_estimate)), into two
parts

$$
\mathrm{err}_X^2 = \frac{1}{n}\mathrm{var}(x) + \frac{1}{n}(\mathrm{cov}(x)-\mathrm{var}(x)),
$$

which equals

<!-- Equation labels as ordinary links -->
<div id="eq:error_estimate_split_up"></div>

$$
\begin{equation}
\frac{1}{n^2}\sum_{k=1}^n (x_k - \bar x_n)^2 +\frac{2}{n^2}\sum_{k<l} (x_k - \bar x_n)(x_l - \bar x_n)
\label{eq:error_estimate_split_up} \tag{17}
\end{equation}
$$

The first term is the same as the error in the uncorrelated case,
Eq. ([16](#eq:error_estimate_uncorrel)). This means that the second
term accounts for the error correction due to correlation between the
measurements. For uncorrelated measurements this second term is zero.

## Statistics, more on computations of errors
Computationally the uncorrelated first term is much easier to treat
efficiently than the second.

$$
\mathrm{var}(x) = \frac{1}{n}\sum_{k=1}^n (x_k - \bar x_n)^2 =
\left(\frac{1}{n}\sum_{k=1}^n x_k^2\right) - \bar x_n^2
$$

We just accumulate separately the values $x^2$ and $x$ for every
measurement $x$ we receive. The correlation term, though, has to be
calculated at the end of the experiment since we need all the
measurements to calculate the cross terms. Therefore, all measurements
have to be stored throughout the experiment.