```python
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:100% !important; }</style>"))
```

<style>.container { width:100% !important; }</style>

# Naive image anomaly detection on Fashion MNIST with Euclidean and Riemannian means, distances, and norms

The goal of this notebook is to evaluate to possibility to achieve anomaly detection in image databases with naive distances to centroids and norms using Euclidean and Riemannian representations. The methods considered being rudimentary, only simple AD setups where the objective is to discriminate between two classes of the Fashion MNIST dataset will be considered. A general approach to embed images into the space of covariance matrices is introduced.

**Table of contents:**

---

[1. Introduction and motivation](#intro)

[1.1. Experiments hyperparameters](#intro1)

---

[2. Analysis/Experiment](#analysis)

[2.1. Dataset loading](#analysis1)

[2.2. Dataset description and preprocessing into covariance matrices](#analysis2)

[2.3. AD with Riemannian distance to Fréchet mean](#analysis3)

[2.4. AD with norm of negated geodesic PCA](#analysis4)

[2.5. AD with norm of negated geodesic PCA with Giotto-TDA preprocessing](#analysis5)

[2.6. Role of Geomstats/Giotto-TDA in the analysis](#analysis6)

---

[3. Benchmark](#benchmark)

[3.1. Euclidean baseline: distance to Euclidean mean](#benchmark1)

[3.2. Euclidean baseline: distance to Euclidean mean after PCA](#benchmark2)

[3.3. Euclidean baseline: Mahalanobis distance to Euclidean mean](#benchmark3)

[3.4. Euclidean baseline: Mahalanobis distance to Euclidean mean after PCA](#benchmark4)

[3.5. Euclidean baseline: norm negated PCA](#benchmark5)

[3.6. Results comparison](#benchmark6)

---

[4. Limitations and perspectives](#limitations)

---

[5. References](#references)

---

```python
# tensorflow not amongst Geomstats’ and Giotto-tda’s requirements.txt
import sys
!{sys.executable} -m pip install tensorflow
!{sys.executable} -m pip install giotto-tda-nightly
```

    Requirement already satisfied: tensorflow in /home/blupon/anaconda3/lib/python3.7/site-packages (2.2.0)
    Requirement already satisfied: wheel>=0.26; python_version >= "3" in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (0.35.1)
    Requirement already satisfied: wrapt>=1.11.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.11.2)
    Requirement already satisfied: protobuf>=3.8.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (3.14.0)
    Requirement already satisfied: numpy<2.0,>=1.16.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.19.2)
    Requirement already satisfied: astunparse==1.6.3 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.6.3)
    Requirement already satisfied: google-pasta>=0.1.8 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (0.2.0)
    Requirement already satisfied: grpcio>=1.8.6 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.31.0)
    WARNING: Keyring is skipped due to an exception: Failed to unlock the collection!
    Collecting scipy==1.4.1; python_version >= "3"
      Using cached scipy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (26.1 MB)
    Requirement already satisfied: h5py<2.11.0,>=2.10.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (2.10.0)
    Requirement already satisfied: absl-py>=0.7.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (0.11.0)
    Requirement already satisfied: opt-einsum>=2.3.2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (3.1.0)
    Requirement already satisfied: keras-preprocessing>=1.1.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.1.2)
    Requirement already satisfied: six>=1.12.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.15.0)
    Requirement already satisfied: tensorboard<2.3.0,>=2.2.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (2.2.2)
    Requirement already satisfied: tensorflow-estimator<2.3.0,>=2.2.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (2.2.0)
    Requirement already satisfied: termcolor>=1.1.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (1.1.0)
    Requirement already satisfied: gast==0.3.3 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorflow) (0.3.3)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (0.4.2)
    Requirement already satisfied: setuptools>=41.0.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (50.3.1.post20201107)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (1.6.0)
    Requirement already satisfied: google-auth<2,>=1.6.3 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (1.27.0)
    Requirement already satisfied: requests<3,>=2.21.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (2.24.0)
    Requirement already satisfied: werkzeug>=0.11.15 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (1.0.1)
    Requirement already satisfied: markdown>=2.6.8 in /home/blupon/anaconda3/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow) (3.3.3)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow) (1.3.0)
    Requirement already satisfied: rsa<5,>=3.1.4; python_version >= "3.6" in /home/blupon/anaconda3/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow) (4.7)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow) (0.2.8)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow) (4.2.1)
    Requirement already satisfied: chardet<4,>=3.0.2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow) (3.0.4)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow) (1.25.11)
    Requirement already satisfied: idna<3,>=2.5 in /home/blupon/anaconda3/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow) (2.10)
    Requirement already satisfied: certifi>=2017.4.17 in /home/blupon/anaconda3/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow) (2020.12.5)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /home/blupon/anaconda3/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow) (2.0.0)
    Requirement already satisfied: oauthlib>=3.0.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow) (3.1.0)
    Requirement already satisfied: pyasn1>=0.1.3 in /home/blupon/anaconda3/lib/python3.7/site-packages (from rsa<5,>=3.1.4; python_version >= "3.6"->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow) (0.4.8)
    Requirement already satisfied: zipp>=0.5 in /home/blupon/anaconda3/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow) (3.4.0)
    Installing collected packages: scipy
      Attempting uninstall: scipy
        Found existing installation: scipy 1.6.3
        Uninstalling scipy-1.6.3:
          Successfully uninstalled scipy-1.6.3
    ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

    We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

    giotto-tda 0.4.0 requires scipy>=1.5.0, but you'll have scipy 1.4.1 which is incompatible.
    giotto-tda-nightly 20210113.12 requires scipy>=1.5.0, but you'll have scipy 1.4.1 which is incompatible.
    geomstats 2.2.2 requires joblib==0.14.1, but you'll have joblib 1.0.1 which is incompatible.
    Successfully installed scipy-1.4.1
    Requirement already satisfied: giotto-tda-nightly in /home/blupon/anaconda3/lib/python3.7/site-packages (20210113.12)
    Requirement already satisfied: joblib>=0.16.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from giotto-tda-nightly) (1.0.1)
    Requirement already satisfied: plotly>=4.8.2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from giotto-tda-nightly) (4.14.3)
    Requirement already satisfied: pyflagser>=0.4.3 in /home/blupon/anaconda3/lib/python3.7/site-packages (from giotto-tda-nightly) (0.4.4)
    Requirement already satisfied: python-igraph>=0.8.2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from giotto-tda-nightly) (0.9.1)
    Requirement already satisfied: ipywidgets>=7.5.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from giotto-tda-nightly) (7.5.1)
    WARNING: Keyring is skipped due to an exception: Failed to unlock the collection!
    Collecting scipy>=1.5.0
      Using cached scipy-1.6.3-cp37-cp37m-manylinux1_x86_64.whl (27.4 MB)
    Requirement already satisfied: numpy>=1.19.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from giotto-tda-nightly) (1.19.2)
    Requirement already satisfied: scikit-learn>=0.23.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from giotto-tda-nightly) (0.23.2)
    Requirement already satisfied: retrying>=1.3.3 in /home/blupon/anaconda3/lib/python3.7/site-packages (from plotly>=4.8.2->giotto-tda-nightly) (1.3.3)
    Requirement already satisfied: six in /home/blupon/anaconda3/lib/python3.7/site-packages (from plotly>=4.8.2->giotto-tda-nightly) (1.15.0)
    Requirement already satisfied: texttable>=1.6.2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from python-igraph>=0.8.2->giotto-tda-nightly) (1.6.3)
    Requirement already satisfied: traitlets>=4.3.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipywidgets>=7.5.1->giotto-tda-nightly) (5.0.5)
    Requirement already satisfied: ipython>=4.0.0; python_version >= "3.3" in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipywidgets>=7.5.1->giotto-tda-nightly) (7.19.0)
    Requirement already satisfied: widgetsnbextension~=3.5.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipywidgets>=7.5.1->giotto-tda-nightly) (3.5.1)
    Requirement already satisfied: nbformat>=4.2.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipywidgets>=7.5.1->giotto-tda-nightly) (5.0.8)
    Requirement already satisfied: ipykernel>=4.5.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipywidgets>=7.5.1->giotto-tda-nightly) (5.3.4)
    Requirement already satisfied: threadpoolctl>=2.0.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from scikit-learn>=0.23.1->giotto-tda-nightly) (2.1.0)
    Requirement already satisfied: ipython-genutils in /home/blupon/anaconda3/lib/python3.7/site-packages (from traitlets>=4.3.1->ipywidgets>=7.5.1->giotto-tda-nightly) (0.2.0)
    Requirement already satisfied: pexpect>4.3; sys_platform != "win32" in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (4.8.0)
    Requirement already satisfied: backcall in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (0.2.0)
    Requirement already satisfied: pickleshare in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (0.7.5)
    Requirement already satisfied: setuptools>=18.5 in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (50.3.1.post20201107)
    Requirement already satisfied: pygments in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (2.7.2)
    Requirement already satisfied: decorator in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (4.4.2)
    Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (3.0.8)
    Requirement already satisfied: jedi>=0.10 in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (0.17.1)
    Requirement already satisfied: notebook>=4.4.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (6.1.4)
    Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets>=7.5.1->giotto-tda-nightly) (3.2.0)
    Requirement already satisfied: jupyter-core in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbformat>=4.2.0->ipywidgets>=7.5.1->giotto-tda-nightly) (4.6.3)
    Requirement already satisfied: jupyter-client in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.5.1->giotto-tda-nightly) (6.1.7)
    Requirement already satisfied: tornado>=4.2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from ipykernel>=4.5.1->ipywidgets>=7.5.1->giotto-tda-nightly) (6.0.4)
    Requirement already satisfied: ptyprocess>=0.5 in /home/blupon/anaconda3/lib/python3.7/site-packages (from pexpect>4.3; sys_platform != "win32"->ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (0.6.0)
    Requirement already satisfied: wcwidth in /home/blupon/anaconda3/lib/python3.7/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (0.2.5)
    Requirement already satisfied: parso<0.8.0,>=0.7.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from jedi>=0.10->ipython>=4.0.0; python_version >= "3.3"->ipywidgets>=7.5.1->giotto-tda-nightly) (0.7.0)
    Requirement already satisfied: Send2Trash in /home/blupon/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (1.5.0)
    Requirement already satisfied: pyzmq>=17 in /home/blupon/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (19.0.2)
    Requirement already satisfied: terminado>=0.8.3 in /home/blupon/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.9.1)
    Requirement already satisfied: prometheus-client in /home/blupon/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.8.0)
    Requirement already satisfied: argon2-cffi in /home/blupon/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (20.1.0)
    Requirement already satisfied: nbconvert in /home/blupon/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (6.0.7)
    Requirement already satisfied: jinja2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (2.11.2)
    Requirement already satisfied: attrs>=17.4.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.5.1->giotto-tda-nightly) (20.3.0)
    Requirement already satisfied: pyrsistent>=0.14.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.17.3)
    Requirement already satisfied: importlib-metadata; python_version < "3.8" in /home/blupon/anaconda3/lib/python3.7/site-packages (from jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.5.1->giotto-tda-nightly) (2.0.0)
    Requirement already satisfied: python-dateutil>=2.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets>=7.5.1->giotto-tda-nightly) (2.8.1)
    Requirement already satisfied: cffi>=1.0.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (1.14.0)
    Requirement already satisfied: nbclient<0.6.0,>=0.5.0 in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.5.1)
    Requirement already satisfied: bleach in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (3.2.1)
    Requirement already satisfied: pandocfilters>=1.4.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (1.4.3)
    Requirement already satisfied: defusedxml in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.6.0)
    Requirement already satisfied: entrypoints>=0.2.2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.3)
    Requirement already satisfied: mistune<2,>=0.8.1 in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.8.4)
    Requirement already satisfied: jupyterlab-pygments in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.1.2)
    Requirement already satisfied: testpath in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.4.4)
    Requirement already satisfied: MarkupSafe>=0.23 in /home/blupon/anaconda3/lib/python3.7/site-packages (from jinja2->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (1.1.1)
    Requirement already satisfied: zipp>=0.5 in /home/blupon/anaconda3/lib/python3.7/site-packages (from importlib-metadata; python_version < "3.8"->jsonschema!=2.5.0,>=2.4->nbformat>=4.2.0->ipywidgets>=7.5.1->giotto-tda-nightly) (3.4.0)
    Requirement already satisfied: pycparser in /home/blupon/anaconda3/lib/python3.7/site-packages (from cffi>=1.0.0->argon2-cffi->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (2.20)
    Requirement already satisfied: async-generator in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (1.10)
    Requirement already satisfied: nest-asyncio in /home/blupon/anaconda3/lib/python3.7/site-packages (from nbclient<0.6.0,>=0.5.0->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (1.4.2)
    Requirement already satisfied: webencodings in /home/blupon/anaconda3/lib/python3.7/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (0.5.1)
    Requirement already satisfied: packaging in /home/blupon/anaconda3/lib/python3.7/site-packages (from bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (20.4)
    Requirement already satisfied: pyparsing>=2.0.2 in /home/blupon/anaconda3/lib/python3.7/site-packages (from packaging->bleach->nbconvert->notebook>=4.4.1->widgetsnbextension~=3.5.0->ipywidgets>=7.5.1->giotto-tda-nightly) (2.4.7)
    Installing collected packages: scipy
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

    We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

    tensorflow 2.2.0 requires scipy==1.4.1; python_version >= "3", but you'll have scipy 1.6.3 which is incompatible.
    geomstats 2.2.2 requires joblib==0.14.1, but you'll have joblib 1.0.1 which is incompatible.
    Successfully installed scipy-1.6.3

```python
import geomstats.backend as gs
from geomstats.learning.kmeans import RiemannianKMeans
import geomstats.geometry.spd_matrices as spd
from geomstats.learning.kmeans import FrechetMean
from geomstats.learning.pca import TangentPCA

from sklearn.metrics import roc_auc_score
from sklearn.covariance import EmpiricalCovariance
from sklearn.decomposition import PCA

from gtda.images import RadialFiltration
from gtda.images import DilationFiltration
from gtda.images import ErosionFiltration
from gtda.images import DensityFiltration
from gtda.images import Binarizer

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

# make sure tensorflow doesn't use GPU to avoid problem of insufficient GPU memory to handle batch size
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '-1'
```

    INFO: Using numpy backend

# <a id='intro'>1. Introduction and motivation</a>

<div style="text-align: justify">
Anomaly detection (AD) in images is a useful tool in a large variety of applications. One can detect anomalies on images to raise an alarm in video surveillance, or to identify flaws for quality control. This AD problem is an active research subject in the machine learning and data processing community. The main difference between AD and classification lies in the supervision and the information available during training. Whereas classification requires representative training data for each mode, AD gathers a single or a set of classes in a unique normal mode and separates the latter from any different data. In order to do this, so-called unsupervised AD only uses normal samples during training to characterize the normality latent distribution, whereas semi-supervised AD additionally takes into account a minority of labelled anomalies to learn the latent boundary. AD remains a difficult task when normality is complex, e.g. multimodal, or when the data provided that defines the latent distribution of normality is not representative, which could lead to a latent boundary easily excluding normal samples in the test phase. Other typical difficulties include potentially infinite diversity when it comes to anomalies and excessive similarity between some normal samples and some anomalies. The following image illustrates what a generic AD setup looks like.
</div>

```python
# show image with python to avoid github error when displaying notebook in browser without jupyter
plt.figure(figsize=(14,9))
plt.imshow(mpimg.imread('multimodal_AD.png'))
```

<div style="text-align: justify">
In this notebook, we explore the performances of naive AD methods with simple distances and norms directly in representation space or after a PCA dimensionality reduction. The objective is to check where naive approaches stand with respect to the very simple AD task of separating two modes of data, i.e. normality is not multimodal and abnormality is not infinitely diverse. This amounts to discriminating between two classes with training data available only for one of the two. The dataset chosen for this experiment is Fashion MNIST. This choice is motivated by the will to practice on a simple enough dataset, avoiding the complexity of CIFAR10 and the lack of multiscale structure in MNIST.
</div>

<br>

<div style="text-align: justify">
Working on distance to centroids and norms asks the question of the representation space on which to compute those features. Following recent research putting forward the relevance of SPD matrices representations, the images will first be transformed into covariance matrices. This enables us to compare the performances of means and distances either defined in an Euclidean setting or a Riemannian setting, the Riemannian setting being provided by the SPD manifold. The Riemannian tools carry the hope that a well chosen manifold makes the distances between samples more relevant with respect to the AD task considered. This translates into the idea that a well chosen manifold is a manifold more adapted to the data distribution.
</div>

<br>

<div style="text-align: justify">
The next figure represents the AD pipelines considered in this notebook, with each notebook subsection implementing the corresponding method indicated on the left.
</div>

```python
# show image with python to avoid github error when displaying notebook in browser without jupyter
plt.figure(figsize=(20,13))
plt.imshow(mpimg.imread('ICLR2021_geomstatsChallenge_pipelines.png'))
```

## <a id='intro1'>1.1. Experiments hyperparameters</a>

```python
results_dic = {}
```

Choice of the Fashion MNIST classes we will separate, we choose 0 (T-shirt) and 2 (Pullover) to obtain a difficult separation problem, since these two classes are quite similar:

```python
class1 = 0
class2 = 2
```

Choice of the number of samples available to compute the mean, defining the normality centroid or reference point:

```python
nsamples = 1000
```

Choice of the covariance matrix type:

```python
cov_type = 'siegel2'
```

Choice of the image axis along which the covariance matrix will be computed:

```python
axis_to_cov = 2
```

Choice of the Giotto-TDA preprocessing filter, and of the binarization threshold for the binarization preceding the filter:

```python
# gfiltertype = 'density'
# gfiltertype = 'dilation'
# gfiltertype = 'erosion'
gfiltertype = 'radial'

binarize_threshold = 0.4
```

Choice of the number of components for PCA:

```python
n_components_PCA = 20
```

# <a id='analysis'>2. Analysis/Experiment</a>

## <a id='analysis1'>2.1. Dataset loading</a>

The so-called normal class seen during training in the unsupersived AD setup is class1, defined in the hyperparameters. The other class, which we try to separate from class1 will only be seen in test.

```python
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()

x_train = x_train/255.
x_test = x_test/255.

filter_class = y_train==class1

x_train1 = x_train[filter_class,:,:]
y_train1 = y_train[filter_class]
x_train1 = x_train1[0:nsamples,:,:]
x_train1 = np.expand_dims(x_train1,axis=-1)

filter_class = y_test==class1
x_test1 = x_test[filter_class]
x_test1 = np.expand_dims(x_test1,axis=-1)

filter_class = y_test==class2
x_test2 = x_test[filter_class]
x_test2 = np.expand_dims(x_test2,axis=-1)
```

## <a id='analysis2'>2.2. Dataset description and preprocessing into covariance matrices</a>

To obtain a relevant SPD matrix to describe an image, transformations of the image will be accumulated, the resulting tensor will be reshaped according to the chosen covariance matrix axis in the hyperparameters, and finally the covariance matrix will be computed. To make this SPD as much relevant as possible, the append mean approach is considered, in order to associate the two first orders representations in the SPD matrix. It can be interpreted as the Riemannian space of non-centered multivariate Gaussian distributions, which has a geometry described by the Siegel metric [[1]](#[1]).

```python
def compute_transformation(x_train):
    x_train=x_train+(np.random.random(x_train.shape)/100)
    xout=[]
    xout.append(x_train)
    for i in range(-3,4,2):
        xtemp=tf.roll(x_train,i,axis=1)
        for j in range(-3,4,2):
            xout.append(tf.roll(xtemp,j,axis=2))
    xout.append(tf.image.flip_left_right(x_train))
    xout.append(tf.image.flip_up_down(x_train))
    xout.append(tf.image.rot90(x_train))
    xout.append(tf.image.rot90(x_train,k=2))
    xout.append(tf.image.rot90(x_train,k=3))
    xout=np.stack(xout,axis=-1)
    return xout
```

```python
def reshape_byaxis(xout,axis=1):
    xout=np.swapaxes(xout,axis,-1)
    return np.reshape(xout,[xout.shape[0],xout.shape[1]*xout.shape[2]*xout.shape[3],xout.shape[4]])
```

In the following function, three possible SPD representations can be computed: a covariance matrix, or two append mean SPD representations combining first and second order moments. As indicated in the code comments, these two append mean approaches stem from the literature [[1]](#[1]) [[2]](#[2]).

<br>

Append mean approach SPD representation with "siegel" option for **compute_cov()** (cf. next notebook cell):

<br>

\begin{equation}
\begin{pmatrix}
\Sigma+ \beta^2 \mu \mu^T & \beta \mu \\
\beta \mu^T & 1
\end{pmatrix}
\end{equation}

<br>

Append mean approach SPD representation with "siegel2" option for **compute_cov()** (cf. next notebook cell):

<br>

\begin{equation}
\begin{pmatrix}
\Sigma+ \beta \mu \mu^T & \beta \mu \\
\beta \mu^T & \beta
\end{pmatrix}
\end{equation}

<br>

In the previous equations, beta is a constant, sigma the covariance matrix, and mu the mean vector.

```python
def compute_cov(xout,option='simple'):
    resC=[]
    if option=='simple':
        for i in range(xout.shape[0]):
            C=np.cov(xout[i,:,:].T)
            resC.append(C)
    elif option=='siegel': # append mean from equation 2 here https://arxiv.org/pdf/1703.06817.pdf
        beta=.3 # Constant recommended in https://arxiv.org/pdf/1703.06817.pdf
        for i in range(xout.shape[0]):
            C=np.cov(xout[i,:,:].T)
            m=np.expand_dims(np.mean(xout[i,:,:],axis=0),axis=-1)
            resC.append(np.concatenate([np.concatenate([C+(beta**2)*(m*m.T),beta*m],axis=1),np.concatenate([beta*m,np.ones([1,1])],axis=0).T]))
    elif option=='siegel2': # append mean from lemma 3.1 here https://core.ac.uk/download/pdf/82584625.pdf
        beta=.3 #Constant recommended in https://arxiv.org/pdf/1703.06817.pdf
        for i in range(xout.shape[0]):
            C=np.cov(xout[i,:,:].T)
            m=np.expand_dims(np.mean(xout[i,:,:],axis=0),axis=-1)
            resC.append(np.concatenate([np.concatenate([C+beta*(m*m.T),beta*m],axis=1),np.concatenate([beta*m,beta*np.ones([1,1])],axis=0).T]))
    return np.array(resC)
```

```python
x_train1_transformations = compute_transformation(x_train1)
x_test1_transformations = compute_transformation(x_test1)
x_test2_transformations = compute_transformation(x_test2)

fig, ax = plt.subplots(2,11,figsize=(20,5))
for transf_index1 in range(2):
    for transf_index2 in range(11):
        ax[transf_index1,transf_index2].imshow(x_train1_transformations[0,:,:,0,transf_index2+11*(transf_index1==1)])
ax[0,5].set_title('Image transformations accumulated in order to compute \na covariance matrix along the horizontal or vertical axis\n', fontsize=20)
```

```python
x_train1_transformations = reshape_byaxis(x_train1_transformations, axis=axis_to_cov)
x_test1_transformations = reshape_byaxis(x_test1_transformations, axis=axis_to_cov)
x_test2_transformations = reshape_byaxis(x_test2_transformations, axis=axis_to_cov)

SPD_train1 = compute_cov(x_train1_transformations, option=cov_type)
SPD_test1 = compute_cov(x_test1_transformations, option=cov_type)
SPD_test2 = compute_cov(x_test2_transformations, option=cov_type)
```

```python
fig, ax = plt.subplots(1,2,figsize=(20,8))
ax[0].imshow(x_train1[3])
ax[0].set_title('Untransformed input image', fontsize=20)
ax[1].imshow(SPD_train1[3])
ax[1].set_title('SPD representation of the image \n(notice the append mean approach illuminating the last column & row)', fontsize=20)

fig, ax = plt.subplots(1,2,figsize=(20,8))
ax[0].imshow(x_test2[7])
ax[0].set_title('Untransformed input image', fontsize=20)
ax[1].imshow(SPD_test2[7])
ax[1].set_title('SPD representation of the image \n(notice the append mean approach illuminating the last column & row)', fontsize=20)
```

## <a id='analysis3'>2.3. AD with Riemannian distance to Fréchet mean</a>

A Riemannian metric must be chosen among those available in Geomstats, in order to compute the Fréchet mean and the distance to this mean

```python
dim = SPD_train1[0].shape[1]
# riem_metric = spd.SPDMetricAffine(dim)
riem_metric = spd.SPDMetricLogEuclidean(dim)
```

Compute Fréchet SPD matrices mean according to previously chosen SPD metric:

```python
FM = FrechetMean(riem_metric).fit(SPD_train1)

plt.imshow(FM.estimate_)
plt.title('Fréchet mean of our normal class training samples')
plt.show()
```

Compute distances of test samples to Fréchet mean:

```python
riemdist_test1 = riem_metric.dist(FM.estimate_,np.array(SPD_test1))
plt.hist(riemdist_test1, alpha=0.2, color='r', bins=100)

riemdist_test2 = riem_metric.dist(FM.estimate_,np.array(SPD_test2))
plt.hist(riemdist_test2, alpha=0.2, color='g', bins=100)

plt.title('Outlier Score for the two classes')
plt.show()
```

ROC AUC achieved:

```python
print(roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([riemdist_test1, riemdist_test2])))
```

    0.7279720000000001

```python
results_dic['dist_frechet_mean'] = roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([riemdist_test1, riemdist_test2]))
```

## <a id='analysis4'>2.4. AD with norm of negated geodesic PCA</a>

Use norm of negated geodesic PCA as detector [[3]](#[3]), with the previously computed Riemannian mean as PCA base point:

```python
tpca = TangentPCA(metric=riem_metric, n_components=n_components_PCA)
tpca = tpca.fit(SPD_train1, base_point=FM.estimate_)

tangent_projected_data_test1 = tpca.transform(SPD_test1)
tangent_projected_data_test2 = tpca.transform(SPD_test2)
```

```python
scores = []
for var in range(0,19):
    scores.append(roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([np.sum(tangent_projected_data_test1[:,var:]**2,axis=1),np.sum(tangent_projected_data_test2[:,var:]**2,axis=1)])))

print("Best ROC AUC is: {} for norm of {} last PCA dimensions".format(max(scores), 20-scores.index(max(scores))))
```

    Best ROC AUC is: 0.8808370000000001 for norm of 14 last PCA dimensions

```python
negPCAscore_test1 = np.sum(tangent_projected_data_test1[:,scores.index(max(scores)):]**2,axis=1)
plt.hist(negPCAscore_test1, alpha=0.2, color='r', bins=100)

negPCAscore_test2 = np.sum(tangent_projected_data_test2[:,scores.index(max(scores)):]**2,axis=1)
plt.hist(negPCAscore_test2, alpha=0.2, color='g', bins=100)

plt.title('Outlier Score for the two classes')
plt.show()
```

```python
results_dic['negated geodesic PCA'] = max(scores)
```

## <a id='analysis5'>2.5. AD with norm of negated geodesic PCA with Giotto-TDA preprocessing</a>

```python
class giotto_preprocessing():
    def __init__(self, x_train, filter_name):
        self.filter_name = filter_name
        if self.filter_name == 'dilation':
            self.filter = DilationFiltration()
        elif self.filter_name == 'erosion':
            self.filter = ErosionFiltration()
        elif self.filter_name == 'density':
            self.filter = DensityFiltration()
        elif self.filter_name == 'radial':
            self.filter = RadialFiltration()
        else:
            raise NoValidGiottoFilterError
        self.filter.fit(x_train)

    def transform(self, x):
        return self.filter.transform(x)
```

Binarize data to be able to apply Giotto-TDA filter:

```python
binarizer = Binarizer(threshold=binarize_threshold)
x_train1 = binarizer.fit_transform(x_train1)

x_test1 = binarizer.transform(x_test1)
x_test2 = binarizer.transform(x_test2)
```

Define Giotto-TDA filter:

```python
gfilter = giotto_preprocessing(x_train1, gfiltertype)
```

Transform train and test data with Giotto-TDA filter and compute new SPD representations:

```python
x_train1_filtered = gfilter.transform(x_train1)

x_test1_filtered = gfilter.transform(x_test1)
x_test2_filtered = gfilter.transform(x_test2)
```

```python
x_train1_transformations_filtered = compute_transformation(x_train1_filtered)
x_test1_transformations_filtered = compute_transformation(x_test1_filtered)
x_test2_transformations_filtered = compute_transformation(x_test2_filtered)

x_train1_transformations_filtered = reshape_byaxis(x_train1_transformations_filtered, axis=axis_to_cov)
x_test1_transformations_filtered = reshape_byaxis(x_test1_transformations_filtered, axis=axis_to_cov)
x_test2_transformations_filtered = reshape_byaxis(x_test2_transformations_filtered, axis=axis_to_cov)

SPD_train1_filtered = compute_cov(x_train1_transformations_filtered, option=cov_type)
SPD_test1_filtered = compute_cov(x_test1_transformations_filtered, option=cov_type)
SPD_test2_filtered = compute_cov(x_test2_transformations_filtered, option=cov_type)
```

```python
fig, ax = plt.subplots(1,2,figsize=(20,8))
ax[0].imshow(x_train1_filtered[3])
ax[0].set_title('Filtered input image', fontsize=20)
ax[1].imshow(SPD_train1_filtered[3])
ax[1].set_title('SPD representation of the filtered image', fontsize=20)

fig, ax = plt.subplots(1,2,figsize=(20,8))
ax[0].imshow(x_test2_filtered[7])
ax[0].set_title('Filtered input image', fontsize=20)
ax[1].imshow(SPD_test2_filtered[7])
ax[1].set_title('SPD representation of the filtered image', fontsize=20)
```

A new Riemannian mean needs to be computed in order to compute the PCA base point:

```python
dim = SPD_train1_filtered[0].shape[1]
# riem_metric = spd.SPDMetricAffine(dim)
riem_metric = spd.SPDMetricLogEuclidean(dim)
```

```python
FM = FrechetMean(riem_metric).fit(SPD_train1_filtered)
FM.estimate_

plt.imshow(FM.estimate_)
plt.title('Fréchet mean of our normal class filtered training samples')
plt.show()
```

```python
tpca = TangentPCA(metric=riem_metric, n_components=n_components_PCA)
tpca = tpca.fit(SPD_train1_filtered, base_point=FM.estimate_)

tangent_projected_fdata_test1 = tpca.transform(SPD_test1_filtered)
tangent_projected_fdata_test2 = tpca.transform(SPD_test2_filtered)
```

```python
scores = []
for var in range(0,19):
    scores.append(roc_auc_score(np.concatenate([np.zeros(len(SPD_test1_filtered)),np.ones(len(SPD_test1_filtered))]),np.concatenate([np.sum(tangent_projected_fdata_test1[:,var:]**2,axis=1),np.sum(tangent_projected_fdata_test2[:,var:]**2,axis=1)])))

print("Best ROC AUC is: {} for norm of {} last PCA dimensions".format(max(scores), 20-scores.index(max(scores))))
```

    Best ROC AUC is: 0.887529 for norm of 13 last PCA dimensions

```python
negPCAscore_test1 = np.sum(tangent_projected_fdata_test1[:,scores.index(max(scores)):]**2,axis=1)
plt.hist(negPCAscore_test1, alpha=0.2, color='r', bins=100)

negPCAscore_test2 = np.sum(tangent_projected_fdata_test2[:,scores.index(max(scores)):]**2,axis=1)
plt.hist(negPCAscore_test2, alpha=0.2, color='g', bins=100)

plt.title('Outlier Score for the two classes')
plt.show()
```

```python
results_dic['negated geodesic PCA with preprocessing filter'] = max(scores)
```

## <a id='analysis6'>2.6. Role of Geomstats/Giotto-TDA in the analysis</a>

<div style="text-align: justify">
The package geomstats allows the AD naive approach, i.e. the distance to the normality reference centroid, to be tested on the SPD manifold on which the data points are projected. The library realizes the most important computations in our experiments, which are the computations of a Riemannian mean and of a Riemannian distance. It also allows us to consider another AD naive approach, i.e. the one where the norm of the last PCA components constitutes an AD score, with its TangentPCA() function, which takes into account the manifold constraint. Finally, Giotto-TDA provides us with a useful preprocessing filter which leads to an improvement in the AD performances.
</div>

# <a id='benchmark'>3. Benchmark</a>

```python
class EuclideanMetric():

    def __init__(self, withMahalanobis=False, train_data=None, pca_reduce=False, n_components=20):
        self.withMahalanobis = withMahalanobis
        self.n_components = n_components
        self.pca_reduce = pca_reduce
        if self.pca_reduce:
            self.pca = PCA(n_components=self.n_components)
        if train_data is None:
            raise NoTrainDataError
        else:
            if self.pca_reduce:
                self.pca = self.pca.fit(train_data.reshape(train_data.shape[0], -1))
            self.centroid = self.init_centroid(train_data)
            if self.withMahalanobis:
                self.empiCov = self.init_Mahalanobis(train_data)
                self.cov = self.empiCov.covariance_

    def init_centroid(self, x):
        x = x.reshape(x.shape[0], -1)
        if self.pca_reduce:
            x = self.pca.transform(x)
        return np.sum(x, axis=0)/x.shape[0]

    def init_Mahalanobis(self, x):
        x = x.reshape(x.shape[0], -1)
        if self.pca_reduce:
            x = self.pca.transform(x)

        empiCov = EmpiricalCovariance()
        empiCov.fit(x)
        plt.imshow(empiCov.covariance_)
        plt.title('Mahalanobis Covariance Matrix')
        plt.show()
        return empiCov

    def dist_to_centroid(self, test_data):
        if self.withMahalanobis:
            return self.Mahalanobis(test_data)
        else:
            return self.Frobenius(test_data)

    def Frobenius(self, x):
        x = x.reshape(x.shape[0], -1)
        if self.pca_reduce:
            x = self.pca.transform(x)
        return np.sqrt(np.sum((np.subtract(x, self.centroid))**2, axis=1))

    def Mahalanobis(self, x):
        x = x.reshape(x.shape[0], -1)
        if self.pca_reduce:
            x = self.pca.transform(x)
        batch_scores = self.empiCov.mahalanobis(x)
        return np.sqrt(batch_scores)
```

## <a id='benchmark1'>3.1. Euclidean baseline: distance to Euclidean mean</a>

```python
baseline_metric = EuclideanMetric(withMahalanobis=False, train_data=SPD_train1, pca_reduce=False)
```

```python
plt.imshow(baseline_metric.centroid.reshape(29,29))
plt.title('Euclidean mean of our normal class training samples')
plt.show()
```

```python
eucldist_test1 = baseline_metric.dist_to_centroid(test_data=np.array(SPD_test1))
plt.hist(eucldist_test1, alpha=0.2, color='r', bins=100)

eucldist_test2 = baseline_metric.dist_to_centroid(test_data=np.array(SPD_test2))
plt.hist(eucldist_test2, alpha=0.2, color='g', bins=100)

plt.title('Outlier Score for two classes')
plt.show()
```

```python
print(roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([eucldist_test1, eucldist_test2])))
```

    0.664795

```python
results_dic['distance to Euclidean mean'] = roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([eucldist_test1, eucldist_test2]))
```

## <a id='benchmark2'>3.2. Euclidean baseline: distance to Euclidean mean after PCA</a>

```python
baseline_metric = EuclideanMetric(withMahalanobis=False, train_data=SPD_train1, pca_reduce=True, n_components=n_components_PCA)
```

```python
eucldist_test1 = baseline_metric.dist_to_centroid(test_data=np.array(SPD_test1))
plt.hist(eucldist_test1, alpha=0.2, color='r', bins=100)

eucldist_test2 = baseline_metric.dist_to_centroid(test_data=np.array(SPD_test2))
plt.hist(eucldist_test2, alpha=0.2, color='g', bins=100)

plt.title('Outlier Score for two classes')
plt.show()
```

```python
print(roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([eucldist_test1, eucldist_test2])))
```

    0.659112

```python
results_dic['distance to Euclidean mean after PCA'] = roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([eucldist_test1, eucldist_test2]))
```

## <a id='benchmark3'>3.3. Euclidean baseline: Mahalanobis distance to Euclidean mean</a>

```python
baseline_metric = EuclideanMetric(withMahalanobis=True, train_data=SPD_train1, pca_reduce=False, n_components=n_components_PCA)
```

```python
eucldist_test1 = baseline_metric.dist_to_centroid(test_data=np.array(SPD_test1))
plt.hist(eucldist_test1, alpha=0.2, color='r', bins=100)

eucldist_test2 = baseline_metric.dist_to_centroid(test_data=np.array(SPD_test2))
plt.hist(eucldist_test2, alpha=0.2, color='g', bins=100)

plt.title('Outlier Score for two classes')
plt.show()
```

```python
print(roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([eucldist_test1, eucldist_test2])))
```

    0.863534

```python
results_dic['Mahalanobis distance to Euclidean mean'] = roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([eucldist_test1, eucldist_test2]))
```

## <a id='benchmark4'>3.4. Euclidean baseline: Mahalanobis distance to Euclidean mean after PCA</a>

```python
baseline_metric = EuclideanMetric(withMahalanobis=True, train_data=SPD_train1, pca_reduce=True, n_components=n_components_PCA)
```

```python
eucldist_test1 = baseline_metric.dist_to_centroid(test_data=np.array(SPD_test1))
plt.hist(eucldist_test1, alpha=0.2, color='r', bins=100)

eucldist_test2 = baseline_metric.dist_to_centroid(test_data=np.array(SPD_test2))
plt.hist(eucldist_test2, alpha=0.2, color='g', bins=100)

plt.title('Outlier Score for two classes')
plt.show()
```

```python
print(roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([eucldist_test1, eucldist_test2])))
```

    0.8198350000000001

```python
results_dic['Mahalanobis distance to Euclidean mean after PCA'] = roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([eucldist_test1, eucldist_test2]))
```

## <a id='benchmark5'>3.5. Euclidean baseline: norm negated PCA</a>

```python
baseline_metric = EuclideanMetric(withMahalanobis=False, train_data=SPD_train1, pca_reduce=True, n_components=n_components_PCA)
```

```python
tangent_projected_data_test1 = baseline_metric.pca.transform(SPD_test1.reshape(SPD_test1.shape[0], -1))
tangent_projected_data_test2 = baseline_metric.pca.transform(SPD_test2.reshape(SPD_test2.shape[0], -1))
```

```python
scores = []
for var in range(0,19):
    scores.append(roc_auc_score(np.concatenate([np.zeros(len(SPD_test1)),np.ones(len(SPD_test2))]),np.concatenate([np.sum(tangent_projected_data_test1[:,var:]**2,axis=1),np.sum(tangent_projected_data_test2[:,var:]**2,axis=1)])))

print("Best ROC AUC is: {} for norm of {} last PCA dimensions".format(max(scores), 20-scores.index(max(scores))))
```

    Best ROC AUC is: 0.8371390000000001 for norm of 4 last PCA dimensions

```python
negPCAscore_test1 = np.sum(tangent_projected_data_test1[:,scores.index(max(scores)):]**2,axis=1)
plt.hist(negPCAscore_test1, alpha=0.2, color='r', bins=100)

negPCAscore_test2 = np.sum(tangent_projected_data_test2[:,scores.index(max(scores)):]**2,axis=1)
plt.hist(negPCAscore_test2, alpha=0.2, color='g', bins=100)

plt.title('Outlier Score for the two classes')
plt.show()
```

```python
results_dic['norm of negated PCA'] = max(scores)
```

## <a id='benchmark6'>3.6. Results comparison</a>

```python
print('ROC AUCs of the method considered:')
for key in results_dic:
    print(results_dic[key], '---->', key)
```

    ROC AUCs of the method considered:
    0.7279720000000001 ----> dist_frechet_mean
    0.8808370000000001 ----> negated geodesic PCA
    0.887529 ----> negated geodesic PCA with preprocessing filter
    0.664795 ----> distance to Euclidean mean
    0.659112 ----> distance to Euclidean mean after PCA
    0.863534 ----> Mahalanobis distance to Euclidean mean
    0.8198350000000001 ----> Mahalanobis distance to Euclidean mean after PCA
    0.8371390000000001 ----> norm of negated PCA

Negated geodesic PCA led to the best ROC AUC performance for the AD task considered. Giotto-TDA preprocessing filter led to a slight performances increase. We also notice the relatively good performances of the Mahalanobis distances in the Euclidean setup.

# <a id='limitations'>4. Limitations and perspectives</a>

# Limitations of this analysis

<div style="text-align: justify">
In this analysis, we projected data on the SPD manifold hoping for a better understanding of the data latent distribution on this mathematical structure. Good performances were obtained with the naive AD method harnessing the SPD representation, but a final conclusion regarding the absolute relevance of the specific manifold choice can not be reached. In order to reach such a conclusion, one would need to compare the projection of the same problem on a wide variety of manifolds. On the one hand, this camparison would allow to detect the best manifold available if any, on the other hand it would help understand the actual contribution of the manifold chosen here.
</div>

<br>

<div style="text-align: justify">
Additionally, the AD problem considered, as well as the competing methods implemented, remain naive in its complexity. Although naive, the methods provide in this specific case relatively interesting performances. A study progressively complexifying the AD problem solved would allow to better understand the limits of such naive methods, and the point after which machine learning with, for example, deep neural networks, is actually needed.
</div>

# Limitation of Geomstats and Giotto-TDA

Trainable SPD representations transformations would have allowed to go further in the Euclidean versus Riemannian comparison, enabling the comparison of more complex models.

# Proposed features for Geomstats and Giotto-TDA

<div style="text-align: justify">
Preprocessing dedicated to SPD matrices, or generating a useful SPD representation for various data types would be an interesting contribution to the machine learning on manifolds toolbox. An implementation of a SPD neural network, or so-called second order neural networks, with the Riemannian gradient and much attention given to the efficiency of the gradient computation, would be welcomed since a widely accepted reference code for this promising model has yet to appear. Moreover, such an implementation would enable more subtle comparisons between AD tasks on a variety of manifolds, to take a step back with respect to the Euclidean methods widely used.
</div>

# <a id='references'>5. References</a>

<a id='[1]'>[1]</a> "A Distance between Multivariate Normal Distributions Based in an Embedding into the Siegel Group" Miquel Calvo and Josep M. Oller

https://core.ac.uk/download/pdf/82584625.pdf

<a id='[2]'>[2]</a> "Second-order Convolutional Neural Networks" Kaicheng Yu and Mathieu Salzmann

https://arxiv.org/pdf/1703.06817.pdf

<a id='[3]'>[3]</a> "Modeling the Distribution of Normal Data in Pre-Trained Deep Features for Anomaly Detection" Oliver Rippel, Patrick Mertens and Dorit Merhof

https://arxiv.org/pdf/2005.14140.pdf

```python

```


# Molecular Dynamics: Lab 2

In part based on [Fortran code from Furio Ercolessi](http://www.fisica.uniud.it/~ercolessi/md/f90/) and the [CHEM126 course of Kalju Khan](http://web.chem.ucsb.edu/~kalju/chem126).

```
from IPython.core.display import HTML
css_file = '../ipython_notebook_styles/ngcmstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Open+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 1000px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1200px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Open Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:900px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Arvo', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Arvo', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Arvo', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Arvo', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Arvo', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>

## Energy minimization

In lab 1 we worked with the Lennard-Jones potential. Other simple potentials include the **harmonic** potential

$$
\begin{equation}
  V = \frac{1}{2} k \left( r - r_{\text{eq}} \right)^2
\end{equation}
$$

with harmonic constant $k$ and equilibrium position $r_{\text{eq}}$, the **Kratzer** potential

$$
\begin{equation}
  V = D_0 \left( \frac{r-r_{\text{eq}}}{r} \right)^2
\end{equation}
$$

with dissociation energy $D_0$, and the **Morse** potential

$$
\begin{equation}
 V = D_0 \left[ 1 - e^{-\alpha (r - r_{\text{eq}})} \right]^2
\end{equation}
$$

where $\alpha$ is the Morse parameter.

### Carbon Monoxide

For example, carbon monoxide (CO) consists of one carbon atom (atomic weight 12.011) and one oxygen atom (atomic weight 15.999) separated by 1.1283 Angstroms. This can be modelled by

1. a harmonic potential with $k = 2743.0$ and $r_{\text{eq}} = 1.1283$ Angstroms, or
2. a Kratzer potential with $D_0 = 258.9$ kcal/mol and the same equilibrium distance, or
3. a Morse potential with $\alpha = 2.302$ Angstroms${}^{-1}$, and the same dissociation energy and equilibrium distance.

Using standard minimization algorithms (e.g. `scipy.optimize.minimize`) show that, for suitable initial guesses, all of the above potentials have minima at the expected location.

By plotting the potentials, understand why the success of the algorithms changes.

Briefly test how changing the potential changes the number of steps required by the algorithm.

Also compute the force $\frac{dV}{dr}$ at the minimum point for each potential to check that it vanishes as expected.

```
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d.axes3d import Axes3D
from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16
rcParams['figure.figsize'] = (12,6)
from scipy.optimize import minimize
```

```

```

### Evolution

Using your velocity-Verlet integrator from lab 1, evolve the CO molecule using the Morse potential. Note that, unlike lab 1, the two particles now have different mass. It is perhaps easiest to scale it so that the carbon particle has mass $1$ and the oxygen mass $1.332029$. Do *not* use periodic boundaries! Try using $\Delta t = 0.001$ up to $t=1$. Start from the equilibrium position and show that it does not evolve. The move the location of the oxygen atom slightly and see how it evolves.

```

```

## Water

The paper of [Praprotnik, Janezic, and Mavri](http://dx.doi.org/10.1021/jp046158d) suggests a potential for a single water molecule (one oxygen atom and two hydrogen with atomic weights $1.008$) of

$$
\begin{equation}
  V = \sum_{l = 1}^2 D_0 \left[ 1 - e^{\alpha \Delta r_{OH_l}} \right]^2 + \frac{1}{2} k_{\theta} \Delta r_{HH}^2 + k_{r \theta} \Delta r_{HH} \left( \Delta r_{OH_1} + \Delta r_{OH_2} \right) + k_{rr} \Delta r_{OH_1} \Delta r_{OH_2}.
\end{equation}
$$

The first term is the Morse potential again, and the other terms are the intra-molecule bond potentials. $\Delta r_{OH_l} = r_{OH_l} - r_{OH_{\text{eq}}}$ is the stretch in the distance between the oxygen atom and the $l^{\text{th}}$ hydrogen atom $r_{OH_l}$ and its equilibrium value, and $\Delta r_{HH} = r_{HH_l} - r_{HH_{\text{eq}}}$ is the stretch in the distance between the hydrogen atoms. The parameter values are

1. $D_0 = 101.9188$ kcal/mol.
2. $\alpha = 2.567$ Angstrom${}^{-1}$
3. $k_{\theta} = 328.645606$ kcal/mol/Angstrom${}^2$
4. $k_{r \theta} = -211.4672$ kcal/mol/Angstrom${}^2$
5. $k_{rr} = 111.70765$ kcal/mol/Angstrom${}^2$
6. $r_{OH_{\text{eq}}} = 1$ Angstrom
7. $r_{HH_{\text{eq}}} = 1.633$ Angstrom

Fix the oxygen atom at the origin and start the hydrogen atoms at $(\pm 0.8, 0.6, 0)$. Use the minimization techniques to find the equilibrium position of the atoms, [comparing against the known structure of water](http://en.wikipedia.org/wiki/Water_model).

**NOTE**: if using `scipy`'s minimization routine, you may have to increase the tolerance as high as $10^{-4}$ to make it converge.

```

```

Using your integrator, evolve the molecule and see how it behaves.

```

```

FEEG6016 Simulation and Modelling/2014/Molecular Dynamics Lab 2.ipynb


```python
"""Tutorial: CEPA0 and CCD"""

__author__    = "Adam S. Abbott"
__credit__    = ["Adam S. Abbott", "Justin M. Turney"]

__copyright__ = "(c) 2014-2017, The Psi4NumPy Developers"
__license__   = "BSD-3-Clause"
__date__      = "2017-05-23"
```

# Introduction
In this tutorial, we will implement the coupled-electron pair approximation (CEPA0) and coupled-cluster doubles (CCD) methods using our spin orbital framework covered in the [previous tutorial](8a_Intro_to_spin_orbital_postHF.ipynb).

### I. Coupled Cluster Theory

In single reference coupled cluster theory, dynamic correlation is acquired by operating an exponential operator on some reference determinant, such as a Hartree-Fock wavefunction, to obtain the coupled cluster wavefunction given by:

\begin{equation}
\mid \mathrm{\Psi_{CC}} \rangle = \exp(\hat{T}) \mid \mathrm{\Phi} \rangle
\end{equation}

where $\hat{T} = T_1 + T_2 + ... + T_n$ is the sum of "cluster operators" which act on our reference wavefunction to excite electrons from occupied ($i, j, k$...) to virtual ($a, b, c$...) orbitals. In second quantization, these cluster operators are expressed as:

\begin{equation}
T_k = \left(\frac{1}{k!}\right)^2 \sum_{\substack{i_1 \ldots i_k \\ a_1 \ldots a_k }} t_{i_1 \ldots i_k}^{a_1 \ldots a_k}   a_{a_1}^{\dagger} \ldots a_{a_k}^{\dagger} a_{i_k} \ldots a_{i_1}
\end{equation}

where $t$ is the $t$-amplitude, and $a^{\dagger}$ and $a$ are creation and annihilation operators.

### II. Coupled Cluster Doubles
For CCD, we only include the doubles cluster operator:

\begin{equation}
\mid \mathrm{\Psi_{CCD}} \rangle = \exp(T_2) \mid \mathrm{\Phi} \rangle
\end{equation}

The CCD Schrödinger equation is

\begin{equation}
\hat{H} \mid \mathrm{\Psi_{CCD}} \rangle = E \mid \mathrm{\Psi_{CCD}}\rangle
\end{equation}

The details will not be covered here, but if we project the CCD Schrödinger equation on the left by our Hartree-Fock reference determinant $ \langle \mathrm{\Phi}\mid $, assuming intermediate normalization $\langle \Phi \mid \mathrm{\Psi_{CCD}} \rangle = 1$, we obtain:

\begin{equation}
 \langle \Phi \mid \hat{H} \space \exp(T_2) \mid \Phi \rangle = E
\end{equation}

which is most easily evaluated with a diagrammatic application of Wick's theorem. Assuming Brillouin's theorem applies (that is, our reference is a Hartree-Fock wavefunction) we obtain:

\begin{equation}
E_{\mathrm{CCD}} = \tfrac{1}{4} \bar{g}_{ij}^{ab} t_{ab}^{ij}
\end{equation}

A somewhat more involved derivation is that of the $t$-amplitudes. These are obtained in a similar fashion to the energy expression, this time projecting the CCD Schrödinger equation on the left by a doubly-excited reference determinant $ \langle\Phi_{ij}^{ab}\mid $:

\begin{equation}
\langle\Phi_{ij}^{ab}\mid \hat{H} \space \exp(T_2) \mid \Phi \rangle
\end{equation}

I will spare you the details of solving this expectation value as well. But, if one evaluates the diagrams via Wick's theorem and simplifies, the $t$-amplitudes are given by:

\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \left( \bar{g}_{ab}^{ij} + \tfrac{1}{2} \bar{g}_{ab}^{cd} t_{cd}^{ij} + \tfrac{1}{2} \bar{g}_{kl}^{ij} t_{ab}^{kl}  + \hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk} - \tfrac{1}{2}\hat{P}_{(a \space / \space b)} \bar{g}_{kl}^{cd} t_{ac}^{ij} t_{bd}^{kl} - \tfrac{1}{2}  \hat{P}^{(i \space / \space j)} \bar{g}_{kl}^{cd} t_{ab}^{ik} t_{cd}^{jl} + \tfrac{1}{4} \bar{g}_{kl}^{cd} t_{cd}^{ij} t_{ab}^{kl} +  \hat{P}^{(i \space / \space j)} \bar{g}_{kl}^{cd} t_{ac}^{ik} t_{bd}^{jl} \right)
\end{equation}

where $(\mathcal{E}_{ab}^{ij})^{-1}$ is the orbital energy denominator, more familiarly known as

\begin{equation}
(\mathcal{E}_{ab}^{ij})^{-1} = \frac{1}{\epsilon_i + \epsilon_j - \epsilon_a - \epsilon_b}
\end{equation}

and $\bar{g}_{pq}^{rs}$ is the antisymmetrized two-electron integral in physicist's notation $\langle pq \mid\mid rs \rangle$. $\hat{P}$ is the *antisymmetric permutation operator*. This operator acts on a term to produce the sum of the permutations of the indicated indices, with an appropriate sign factor. Its effect is best illustrated by an example. Consider the fourth term, which is really four terms in one.

$\hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk}$ produces:

1. The original: $ \quad \bar{g}_{ak}^{ic} t_{bc}^{jk} \\ $

2. Permuation of $a$ and $b$: $ \quad  \textrm{-} \bar{g}_{bk}^{ic} t_{ac}^{jk} \\ $

3. Permuation of $i$ and $j$: $ \quad \, \, \textrm{-} \bar{g}_{ak}^{jc} t_{bc}^{ik} \\ $

4. Permuation of $a$ and $b$, $i$ and $j$: $ \quad \bar{g}_{bk}^{jc} t_{ac}^{ik} \\ $

Note that each permutation adds a sign change. This shorthand notation keeps the equation in a more manageable form.

Since the $t$-amplitudes and the energy depend on $t$-amplitudes, we must iteratively solve these equations until they reach self consistency, and the energy converges to some threshold.

### III. Retrieving MP2 and CEPA0 from the CCD equations
It is interesting to note that if we only consider the first term of the expression for the doubles amplitude $t_{ab}^{ij}$ and plug it into the energy expression, we obtain the MP2 energy expression:

\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \bar{g}_{ab}^{ij}
\end{equation}

\begin{equation}
E_{\mathrm{MP2}} = \tfrac{1}{4}  \bar{g}_{ij}^{ab} t_{ab}^{ij}  = \tfrac{1}{4} \bar{g}_{ij}^{ab} \bar{g}_{ab}^{ij}   (\mathcal{E}_{ab}^{ij})^{-1}
\end{equation}

Furthermore, if we leave out the quadratic terms in the CCD amplitude equation (terms containing two $t$-amplitudes), we obtain the coupled electron-pair approximation (CEPA0):
\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \left( \bar{g}_{ab}^{ij} + \tfrac{1}{2} \bar{g}_{ab}^{cd} t_{cd}^{ij} + \tfrac{1}{2} \bar{g}_{kl}^{ij} t_{ab}^{kl}  + \hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk} \right)
\end{equation}

The CEPA0 energy expression is identical:

\begin{equation}
E_{\mathrm{CEPA0}} = \tfrac{1}{4} \bar{g}_{ij}^{ab} t_{ab}^{ij}
\end{equation}

Using our spin orbital setup for the MO coefficients, orbital energies, and two-electron integrals used in the [previous tutorial](8a_Intro_to_spin_orbital_postHF.ipynb), we are equipped to program the expressions for the CEPA0 and CCD correlation energy.

### Implementation: CEPA0 and CCD
As usual, we import Psi4 and NumPy, and set the appropriate options.

```python
# ==> Import statements & Global Options <==
import psi4
import numpy as np

psi4.set_memory(int(2e9))
numpy_memory = 2
psi4.core.set_output_file('output.dat', False)
```

```python
# ==> Molecule & Psi4 Options Definitions <==
mol = psi4.geometry("""
0 1
O
H 1 1.1
H 1 1.1 2 104
symmetry c1
""")

psi4.set_options({'basis':        '6-31g',
                  'scf_type':     'pk',
                  'reference':    'rhf',
                  'mp2_type':     'conv',
                  'e_convergence': 1e-8,
                  'd_convergence': 1e-8})
```

Note that since we are using a spin orbital setup, we are free to use any Hartree-Fock reference we want. Here we choose RHF. For convenience, we let Psi4 take care of the Hartree-Fock procedure, and return the wavefunction object.

```python
# Get the SCF wavefunction & energies
scf_e, scf_wfn = psi4.energy('scf', return_wfn=True)
```

Load in information about the basis set and orbitals using MintsHelper and the wavefunction:

```python
mints = psi4.core.MintsHelper(scf_wfn.basisset())
nbf = mints.nbf()           # number of basis functions
nso = 2 * nbf               # number of spin orbitals
nalpha = scf_wfn.nalpha()   # number of alpha electrons
nbeta = scf_wfn.nbeta()     # number of beta electrons
nocc = nalpha + nbeta       # number of occupied orbitals
nvirt = 2 * nbf - nocc      # number of virtual orbitals
```

Spin-block our MO coefficients and two-electron integrals, just like in the spin orbital MP2 code:

```python
Ca = np.asarray(scf_wfn.Ca())
Cb = np.asarray(scf_wfn.Cb())
C = np.block([
             [      Ca           ,   np.zeros_like(Cb) ],
             [np.zeros_like(Ca)  ,          Cb         ]
            ])

# Result: | Ca  0 |
#         | 0   Cb|

```

```python
# Get the two electron integrals using MintsHelper
I = np.asarray(mints.ao_eri())

def spin_block_tei(I):
    """
    Function that spin blocks two-electron integrals
    Using np.kron, we project I into the space of the 2x2 identity, tranpose the result
    and project into the space of the 2x2 identity again. This doubles the size of each axis.
    The result is our two electron integral tensor in the spin orbital form.
    """
    identity = np.eye(2)
    I = np.kron(identity, I)
    return np.kron(identity, I.T)

# Spin-block the two electron integral array
I_spinblock = spin_block_tei(I)

```

Convert two-electron integrals to antisymmetrized physicist's notation:

```python
# Converts chemist's notation to physicist's notation, and antisymmetrize
# (pq | rs) ---> <pr | qs>
# Physicist's notation
tmp = I_spinblock.transpose(0, 2, 1, 3)
# Antisymmetrize:
# <pr||qs> = <pr | qs> - <pr | sq>
gao = tmp - tmp.transpose(0, 1, 3, 2)

```

Obtain the orbital energies, append them, and sort the columns of our MO coefficient matrix according to the increasing order of orbital energies.

```python
# Get orbital energies
eps_a = np.asarray(scf_wfn.epsilon_a())
eps_b = np.asarray(scf_wfn.epsilon_b())
eps = np.append(eps_a, eps_b)

# Before sorting the orbital energies, we can use their current arrangement to sort the columns
# of C. Currently, each element i of eps corresponds to the column i of C, but we want both
# eps and columns of C to be in increasing order of orbital energies

# Sort the columns of C according to the order of increasing orbital energies
C = C[:, eps.argsort()]

# Sort orbital energies in increasing order
eps = np.sort(eps)

```

Finally, we transform our two-electron integrals to the MO basis. Here, we denote the integrals as `gmo` to differentiate from the chemist's notation integrals `I_mo`.

```python
# Transform gao, which is the spin-blocked 4d array of physicist's notation,
# antisymmetric two-electron integrals, into the MO basis using MO coefficients
gmo = np.einsum('pQRS, pP -> PQRS',
      np.einsum('pqRS, qQ -> pQRS',
      np.einsum('pqrS, rR -> pqRS',
      np.einsum('pqrs, sS -> pqrS', gao, C), C), C), C)

```

Construct the 4-dimensional array of orbital energy denominators:

```python
# Define slices, create 4 dimensional orbital energy denominator tensor
n = np.newaxis
o = slice(None, nocc)
v = slice(nocc, None)
e_abij = 1 / (-eps[v, n, n, n] - eps[n, v, n, n] + eps[n, n, o, n] + eps[n, n, n, o])
```

We now have everything we need to construct our $t$-amplitudes and iteratively solve for our CEPA0 and CCD energy. To build the $t$-amplitudes, we first construct an empty 4-dimensional array to store them.

```python
# Create space to store t amplitudes
t_amp = np.zeros((nvirt, nvirt, nocc, nocc))

```

# Implementation: CEPA0
First we will program CEPA0. Recall the expression for the $t$-amplitudes:

\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \left( \bar{g}_{ab}^{ij} + \tfrac{1}{2} \bar{g}_{ab}^{cd} t_{cd}^{ij} + \tfrac{1}{2} \bar{g}_{kl}^{ij} t_{ab}^{kl}  + \hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk} \right)
\end{equation}

These terms translate naturally into code using NumPy's `einsum` function. To access only the occupied and virtual indices of `gmo` we use our slices defined above. The permutation operator terms can be easily obtained by transposing the original result accordingly. To construct each iteration's $t$-amplitude:

~~~python
mp2    = gmo[v, v, o, o]
cepa1  = (1 / 2) * np.einsum('abcd, cdij -> abij', gmo[v, v, v, v], t_amp)
cepa2  = (1 / 2) * np.einsum('klij, abkl -> abij', gmo[o, o, o, o], t_amp)
cepa3a = np.einsum('akic, bcjk -> abij', gmo[v, o, o, v], t_amp)
cepa3b = -cepa3a.transpose(1, 0, 2, 3)
cepa3c = -cepa3a.transpose(0, 1, 3, 2)
cepa3d =  cepa3a.transpose(1, 0, 3, 2)
cepa3  =  cepa3a + cepa3b + cepa3c + cepa3d

t_amp_new = e_abij * (mp2 + cepa1 + cepa2 + cepa3)
~~~

To evaluate the energy, $E_{\mathrm{CEPA0}} = \tfrac{1}{4} \bar{g}_{ij}^{ab} t_{ab}^{ij}$,

~~~python
E_CEPA0 = (1 / 4) * np.einsum('ijab, abij ->', gmo[o, o, v, v], t_amp_new)
~~~

Putting it all together, we initialize the energy, set the max iterations, and iterate the energy until it converges to our convergence criterion:

```python
# Initialize energy
E_CEPA0 = 0.0

MAXITER = 50

for cc_iter in range(MAXITER + 1):
    E_old = E_CEPA0

    # Collect terms
    mp2    = gmo[v, v, o, o]
    cepa1  = (1 / 2) * np.einsum('abcd, cdij -> abij', gmo[v, v, v, v], t_amp)
    cepa2  = (1 / 2) * np.einsum('klij, abkl -> abij', gmo[o, o, o, o], t_amp)
    cepa3a = np.einsum('akic, bcjk -> abij', gmo[v, o, o, v], t_amp)
    cepa3b = -cepa3a.transpose(1, 0, 2, 3)
    cepa3c = -cepa3a.transpose(0, 1, 3, 2)
    cepa3d =  cepa3a.transpose(1, 0, 3, 2)
    cepa3  =  cepa3a + cepa3b + cepa3c + cepa3d

    # Update t amplitude
    t_amp_new = e_abij * (mp2 + cepa1 + cepa2 + cepa3)

    # Evaluate Energy
    E_CEPA0 = (1 / 4) * np.einsum('ijab, abij ->', gmo[o, o, v, v], t_amp_new)
    t_amp = t_amp_new
    dE = E_CEPA0 - E_old
    print('CEPA0 Iteration %3d: Energy = %4.12f dE = %1.5E' % (cc_iter, E_CEPA0, dE))

    if abs(dE) < 1.e-8:
        print("\nCEPA0 Iterations have converged!")
        break

    if (cc_iter == MAXITER):
        psi4.core.clean()
        raise Exception("\nMaximum number of iterations exceeded.")

print('\nCEPA0 Correlation Energy: %5.15f' % (E_CEPA0))
print('CEPA0 Total Energy: %5.15f' % (E_CEPA0 + scf_e))
```

    CEPA0 Iteration   0: Energy = -0.142119840297 dE = -1.42120E-01
    CEPA0 Iteration   1: Energy = -0.142244391285 dE = -1.24551E-04
    CEPA0 Iteration   2: Energy = -0.146403555998 dE = -4.15916E-03
    CEPA0 Iteration   3: Energy = -0.147737944877 dE = -1.33439E-03
    CEPA0 Iteration   4: Energy = -0.148357998670 dE = -6.20054E-04
    CEPA0 Iteration   5: Energy = -0.148640319451 dE = -2.82321E-04
    CEPA0 Iteration   6: Energy = -0.148774677657 dE = -1.34358E-04
    CEPA0 Iteration   7: Energy = -0.148840007370 dE = -6.53297E-05
    CEPA0 Iteration   8: Energy = -0.148872388063 dE = -3.23807E-05
    CEPA0 Iteration   9: Energy = -0.148888687541 dE = -1.62995E-05
    CEPA0 Iteration  10: Energy = -0.148897003541 dE = -8.31600E-06
    CEPA0 Iteration  11: Energy = -0.148901297946 dE = -4.29440E-06
    CEPA0 Iteration  12: Energy = -0.148903540421 dE = -2.24248E-06
    CEPA0 Iteration  13: Energy = -0.148904723684 dE = -1.18326E-06
    CEPA0 Iteration  14: Energy = -0.148905354221 dE = -6.30537E-07
    CEPA0 Iteration  15: Energy = -0.148905693366 dE = -3.39145E-07
    CEPA0 Iteration  16: Energy = -0.148905877390 dE = -1.84023E-07
    CEPA0 Iteration  17: Energy = -0.148905978068 dE = -1.00678E-07
    CEPA0 Iteration  18: Energy = -0.148906033572 dE = -5.55039E-08
    CEPA0 Iteration  19: Energy = -0.148906064388 dE = -3.08158E-08
    CEPA0 Iteration  20: Energy = -0.148906081607 dE = -1.72194E-08
    CEPA0 Iteration  21: Energy = -0.148906091285 dE = -9.67815E-09

    CEPA0 Iterations have converged!

    CEPA0 Correlation Energy: -0.148906091285338
    CEPA0 Total Energy: -76.101435136027135

Since `t_amp` is initialized to zero, the very first iteration should be the MP2 correlation energy. We can check the final CEPA0 energy with Psi4. The method is called `lccd`, or linear CCD, since CEPA0 omits the terms with two cluster amplitudes.

```python
psi4.driver.p4util.compare_values(psi4.energy('lccd'), E_CEPA0 + scf_e, 6, 'CEPA0 Energy')
```

    	CEPA0 Energy......................................................PASSED

    True

# Implementation: CCD

To code CCD, we only have to add in the last four terms in our expression for the $t$-amplitudes:

\begin{equation}
t_{ab}^{ij} = (\mathcal{E}_{ab}^{ij})^{-1} \left( \bar{g}_{ab}^{ij} + \tfrac{1}{2} \bar{g}_{ab}^{cd} t_{cd}^{ij} + \tfrac{1}{2} \bar{g}_{kl}^{ij} t_{ab}^{kl}  + \hat{P}_{(a \space / \space b)}^{(i \space / \space j)} \bar{g}_{ak}^{ic} t_{bc}^{jk} - \underline{\tfrac{1}{2}\hat{P}_{(a \space / \space b)} \bar{g}_{kl}^{cd} t_{ac}^{ij} t_{bd}^{kl} - \tfrac{1}{2}  \hat{P}^{(i \space / \space j)} \bar{g}_{kl}^{cd} t_{ab}^{ik} t_{cd}^{jl} + \tfrac{1}{4} \bar{g}_{kl}^{cd} t_{cd}^{ij} t_{ab}^{kl} +  \hat{P}^{(i \space / \space j)} \bar{g}_{kl}^{cd} t_{ac}^{ik} t_{bd}^{jl}} \right)
\end{equation}

which we readily translate into `einsum`'s:

~~~python
ccd1a  = np.einsum('klcd, acij, bdkl -> abij', gmo[o, o, v, v], t_amp, t_amp)
ccd1b  = -ccd1a.transpose(1, 0, 2, 3)
ccd1   = -(1 / 2) * (ccd1a + ccd1b)

ccd2a  = np.einsum('klcd, abik, cdjl -> abij', gmo[o, o, v, v], t_amp, t_amp)
ccd2b  = -ccd2a.transpose(0, 1, 3, 2)
ccd2   = -(1 / 2) * (ccd2a + ccd2b)

ccd3   =  (1 / 4) * np.einsum('klcd, cdij, abkl -> abij', gmo[o, o, v, v], t_amp, t_amp)

ccd4a  = np.einsum('klcd, acik, bdjl -> abij', gmo[o, o, v, v], t_amp, t_amp)
ccd4b  = -ccd4a.transpose(0, 1, 3, 2)
ccd4   = (ccd4a + ccd4b)
~~~

and the energy expression is identical to CEPA0:
\begin{equation}
E_{CCD } = \tfrac{1}{4} \bar{g}_{ij}^{ab} t_{ab}^{ij}
\end{equation}

Adding the above terms to our CEPA0 code will compute the CCD correlation energy (may take a minute or two to run):

```python
# Create space to store t amplitudes
t_amp = np.zeros((nvirt, nvirt, nocc, nocc))

# Initialize energy
E_CCD = 0.0

for cc_iter in range(1, MAXITER + 1):
    E_old = E_CCD

    # Collect terms
    mp2    = gmo[v, v, o, o]
    cepa1  = (1 / 2) * np.einsum('abcd, cdij -> abij', gmo[v, v, v, v], t_amp)
    cepa2  = (1 / 2) * np.einsum('klij, abkl -> abij', gmo[o, o, o, o], t_amp)
    cepa3a = np.einsum('akic, bcjk -> abij', gmo[v, o, o, v], t_amp)
    cepa3b = -cepa3a.transpose(1, 0, 2, 3)
    cepa3c = -cepa3a.transpose(0, 1, 3, 2)
    cepa3d =  cepa3a.transpose(1, 0, 3, 2)
    cepa3  =  cepa3a + cepa3b + cepa3c + cepa3d

    ccd1a  = np.einsum('klcd, acij, bdkl -> abij', gmo[o, o, v, v], t_amp, t_amp)
    ccd1b  = -ccd1a.transpose(1, 0, 2, 3)
    ccd1   = -(1 / 2) * (ccd1a + ccd1b)

    ccd2a  = np.einsum('klcd, abik, cdjl -> abij', gmo[o, o, v, v], t_amp, t_amp)
    ccd2b  = -ccd2a.transpose(0, 1, 3, 2)
    ccd2   = -(1 / 2) * (ccd2a + ccd2b)

    ccd3   =  (1 / 4) * np.einsum('klcd, cdij, abkl -> abij', gmo[o, o, v, v], t_amp, t_amp)

    ccd4a  = np.einsum('klcd, acik, bdjl -> abij', gmo[o, o, v, v], t_amp, t_amp)
    ccd4b  = -ccd4a.transpose(0, 1, 3, 2)
    ccd4   = (ccd4a + ccd4b)

    # Update Amplitude
    t_amp_new = e_abij * (mp2 + cepa1 + cepa2 + cepa3 + ccd1 + ccd2 + ccd3 + ccd4)

    # Evaluate Energy
    E_CCD = (1 / 4) * np.einsum('ijab, abij ->', gmo[o, o, v, v], t_amp_new)
    t_amp = t_amp_new
    dE = E_CCD - E_old
    print('CCD Iteration %3d: Energy = %4.12f dE = %1.5E' % (cc_iter, E_CCD, dE))

    if abs(dE) < 1.e-8:
        print("\nCCD Iterations have converged!")
        break

    if (cc_iter == MAXITER):
        psi4.core.clean()
        raise Exception("\nMaximum number of iterations exceeded.")

print('\nCCD Correlation Energy: %5.15f' % (E_CCD))
print('CCD Total Energy: %5.15f' % (E_CCD + scf_e))

```

Unfortunately, Psi4 does not have a CCD code to compare this to. However, Psi4 does have Bruekner CCD, an orbital-optimized variant of CCD. We can qualitatively compare our energies to this energy. The Bruekner-CCD energy should be a little lower than our CCD energy due to the orbital optimization procedure.

```python
psi4_bccd = psi4.energy('bccd', ref_wfn = scf_wfn)
print('\nPsi4 BCCD Correlation Energy:     ', psi4_bccd - scf_e)
print('Psi4 BCCD Total Energy:     ', psi4_bccd)
```

    Psi4 BCCD Correlation Energy:      -0.1492076631396344
    Psi4 BCCD Total Energy:      -76.10173670788143

## References

1. Modern review of coupled-cluster theory, included diagrammatic derivations of the CCD equations:
	> [[Bartlett and Musial:2007](https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.79.291)] Rodney J. Bartlett and Monika Musial,  "Coupled-cluster theory in quantum chemistry" *Rev. Mod. Phys.* **79**, 291 (2007)

2. Background on CEPA:
    >Kutzelnigg, Werner 1977 *Methods of Electronic Structure Theory* ed. H. F. Schaefer III (Plenum, New York), p 129

3. More CEPA:
    > [Koch and Kutzelnigg:1981](https://link.springer.com/article/10.1007/BF00553396) S. Koch and W. Kutzelnigg, *Theor. Chim. Acta* **59**, 387 (1981).

4. Original CCD Paper:
    > [Čížek:1996](http://aip.scitation.org/doi/abs/10.1063/1.1727484) Jiří Čížek, "On the Correlation Problem in Atomic and Molecular Systems. Calculation of Wavefunction Components in Ursell‐Type Expansion Using Quantum‐Field Theoretical Methods" *J. Chem. Phys* **45**, 4256 (1966)

5. Useful notes on diagrams applied to post-HF methods:
    > A. V. Copan, "Diagram notation" accessed with https://github.com/CCQC/chem-8950/tree/master/2017