# 03 Hartree-Fock Closed-Shell SCF Procedure

The purpose of this project is to provide a deeper understanding of (*restricted*) Hartree-Fock theory by demonstrating a simple implementation of the self-consistent-field (SCF) method. The theoretical background can be found in Ch. 3 of the text by Szabo and Ostlund or in the [nice set of on-line notes](http://vergil.chemistry.gatech.edu/notes/hf-intro/hf-intro.html) written by David Sherrill. A concise set of instructions for this project may be found [here](https://github.com/CrawfordGroup/ProgrammingProjects/blob/master/Project%2303/project3-instructions.pdf).

Original authors (Crawford, et al.) thank Dr. Yukio Yamaguchi of the University of Georgia for the original version of this project.

This project could be a challenging one. Although it is somehow easy to achieve each steps, the number of steps is quite large. You may well possibly have done all tasks, but confuse on how and why iterative SCF could be written in this way. You are encouraged to review this project once you've done all steps.

The test case used in the following discussion is for a water molecule with a bond-length of $1.1 {\buildrel_{\circ} \over{\mathsf{A}}}$ and a bond angle of $104.0 {}^\circ$ with an STO-3G basis set. The input to the project consists of the nuclear repulsion energy ({download}`input/h2o/STO-3G/enuc.dat`) and pre-computed sets of one- and two-electron integrals: overlap integrals ({download}`input/h2o/STO-3G/s.dat`), kinetic-energy integrals ({download}`input/h2o/STO-3G/t.dat`), nuclear-attraction integrals ({download}`input/h2o/STO-3G/v.dat`), electron-electron repulsion integrals ({download}`input/h2o/STO-3G/eri.dat`).

````{admonition} Multiple ways to achieve a task
:class: warning

From this project, it could be multiple ways to achieve a task. What *multiple* means here is that one could
- either read integrals from input file, or from PySCF's integral interface;
- either do (high-dimension) tensor contraction by extremely powerful `numpy.einsum` ([NumPy API](https://numpy.org/doc/stable/reference/generated/numpy.einsum.html)), or by simply for-looping, or hybrid for-looping and 2-dimension matrix manipulation;
- format outputs by his own style (with most important values unchanged to the reference solution output);
- etc.

Multiple approaches could arrive to the almost same solution, and every approach is appreciated. The reader may develop his/hers own coding style. However, as author of the documentation, there's still some partiality for using PySCF's integral interface and using `numpy.einsum`.

````

In this project, using module `pyscf.scf` is prohibited. Using `pyscf.gto` is permitted. The structure of this project is slightly changed compared to the [original project 03](https://github.com/CrawfordGroup/ProgrammingProjects/tree/master/Project%2303).

```python
# Following os.chdir code is only for thebe (live code), since only in thebe default directory is /home/jovyan
import os
if os.getcwd().split("/")[-1] != "Project_03":
    os.chdir("source/Project_03")
from solution_03 import Molecule as SolMol
# Following code is called for numpy array pretty printing
from pyscf import gto
import numpy as np
from scipy.linalg import fractional_matrix_power
from pprint import pp  # Data Pretty Printer, as it says
from typing import Dict, Tuple  # only used in type infer
np.set_printoptions(precision=7, linewidth=120, suppress=True)
```

```python
sol_mole = SolMol()
sol_mole.construct_from_dat_file("input/h2o/STO-3G/geom.dat")
sol_mole.path_dict = {
    "enuc": "input/h2o/STO-3G/enuc.dat",
    "s": "input/h2o/STO-3G/s.dat",
    "t": "input/h2o/STO-3G/t.dat",
    "v": "input/h2o/STO-3G/v.dat",
    "eri": "input/h2o/STO-3G/eri.dat",
    "mux": "input/h2o/STO-3G/mux.dat",
    "muy": "input/h2o/STO-3G/muy.dat",
    "muz": "input/h2o/STO-3G/muz.dat",
}
```

````{admonition} Notation Convention
:class: warning, dropdown

It is important to make notation convention clear at the very beginning, or otherwise one may could have taken the wrong comprehension all the time reading article or documentation even without recognizing that. Here we make some essential, but not all, notations that we use in the documents thereafter. Meaning of some notations to some readers maybe not clear or familiar. These notations would be discussed in later sections or documents.

- $\mu, \nu, \kappa, \lambda$: Atomic orbitals. These notations are different to Szabo and Ostlund's $\mu, \nu, \sigma, \lambda$. Decision of changing $\sigma$ to $\kappa$ based on the convention that $\sigma$ is more abused to be notation of orbital spin.
- $i, j, k, l, ...$: Occupied orbitals. These notations are more accepted nowadays, which are different to Szabo and Ostlund's $a, b, c, ...$.
- $a, b, c, d, ...$: Virtual orbitals (un-occupied orbitals). The same reason to above.
- $p, q, r, s, m$: All accepted orbitals.
- $n_\mathrm{occ}, n_\mathrm{vir}, n_\mathrm{AO}, n_\mathrm{MO}$: Number of occupied, virtual, atomic and molecular orbitals, respectively.
- $A, B, C, M$: Atom subscripts. These notations are different to the previous projects' $i, j, k, ...$
- $\boldsymbol{r}, \boldsymbol{R}_A$: Coordinate of electron and nucleus of atom $A$, respectively.
- $Z_A$: Atomic charge of atom $A$.

In the documentation, we will not learn frozen orbitals. So division of occupied and virtual is sufficient.

````

## Molecule Object Initialization

In this project, we may use the updated `Molecule` initialization. Since these are technical details, we toggle the code and explanation below.

```python
class Molecule:

    def __init__(self):
        self.atom_charges = NotImplemented  # type: np.ndarray
        self.atom_coords = NotImplemented   # type: np.ndarray
        self.natm = NotImplemented          # type: int
        self.mol = NotImplemented           # type: gto.Mole
        self.path_dict = NotImplemented     # type: Dict[str, str]
        self.nao = NotImplemented           # type: int
        self.charge = 0                     # type: int
        self.nocc = NotImplemented          # type: int

    def construct_from_dat_file(self, file_path: str):
        # Same to Project 01
        with open(file_path, "r") as f:
            dat = np.array([line.split() for line in f.readlines()][1:])
            self.atom_charges = np.array(dat[:, 0], dtype=float).astype(int)
            self.atom_coords = np.array(dat[:, 1:4], dtype=float)
            self.natm = self.atom_charges.shape[0]

mole = Molecule()
mole.construct_from_dat_file("input/h2o/STO-3G/geom.dat")
```

````{toggle} Toggled explanations

The meaning of attributes of `Molecule` are:

- `atom_charges`: Array to store atomic charges
- `atom_coords`: Matrix to store coordinates of atoms
- `natm`: Number of atoms
- `mol`: `pyscf.gto.Mole` instance, which describes molecular and basis information; *only used in PySCF approaches*
- `path_dict`: Dictionary of file path which containing integral, energy, ...; *only used in reading-input approaches*
- `nao`: Number of atomic orbitals $n_\mathrm{AO}$
- `charge`: Molecular charge ($\mathsf{NH_4^+}$ contains +1 charge and $\mathsf{NO_3^-}$ contains -1 charge); in most cases we just use neutral molecules
- `nocc`: Number of occupied orbitals $n_\mathrm{occ}$

Readers may be familier with the first three attributes which we have extensively used in [Project 01](../Project_01/Project_01.ipynb) and [Project 02](../Project_02/Project_02.ipynb). One may call `Molecule.construct_from_dat_file` to initialize these three attributes. For demonstration, we use STO-3G water molecule which is instantiated as variable `mole`.

````

````{admonition} Extra initialization: PySCF approach
:class: dropdown

In PySCF, molecule and basis information is stored in `pyscf.gto.Mole` ([PySCF API](http://pyscf.org/pyscf/gto.html)) instance. For example, the water molecule and basis STO-3G (the same to de demonstrated molecule) could be instantiated by

```python
mol = gto.Mole()
mol.unit = "Bohr"
mol.atom = """
8   0.000000000000  -0.143225816552   0.000000000000
1   1.638036840407   1.136548822547  -0.000000000000
1  -1.638036840407   1.136548822547  -0.000000000000
"""
mol.basis = "STO-3G"
mol.build()
```

Though the molecule can be manually instantiated, we still need a method function in `Molecule`, using existing `mole.atom_charges` and `mole.atom_coords` and external basis `"STO-3G"`, to build the `pyscf.gto.Mole` instance and store in `mole.mol` attribute.

Note that we just discuss closed-shell situation, so we set `mol.spin = 0`. The convention of molecular spin in different softwares might be quite deviated. In PySCF, `mol.spin` means number of alpha electrons minus beta electrons.

Class `pyscf.gto.Mole` can be extremely useful and convenient for this documentation, as almost all electron integrals can be called by method function `pyscf.gto.Mole.intor` in only one short line of code.

````

**Implementation**

Reader should fill all `NotImplementedError` in the following code:

```python
def obtain_mol_instance(mole: Molecule, basis: str, verbose=0):
    # Input: Basis string information
    #        (for advanced users, see PySCF API, this could be dictionary instead of string)
    # Attribute modification: Obtain `pyscf.gto.Mole` instance `mole.mol`
    mol = gto.Mole()
    mol.unit = "Bohr"
    raise NotImplementedError("About 2~20 lines of code, where `mol.atom` and `mol.basis` need to be defined")
    mol.charge = mole.charge
    mol.spin = 0
    mol.verbose = verbose
    mole.mol = mol.build()

Molecule.obtain_mol_instance = obtain_mol_instance
```

**Solution**

```python
sol_mole.obtain_mol_instance(basis="STO-3G")
print("=== Coordinate ===")
print(sol_mole.mol.atom_coords())
print("=== Atomic Charges ===")
print(sol_mole.mol.atom_charges())
print("=== Basis Detail ===")
pp(sol_mole.mol._basis)
```

    === Coordinate ===
    [[ 0.        -0.1432258  0.       ]
     [ 1.6380368  1.1365488  0.       ]
     [-1.6380368  1.1365488  0.       ]]
    === Atomic Charges ===
    [8 1 1]
    === Basis Detail ===
    {'H': [[0,
            [3.42525091, 0.15432897],
            [0.62391373, 0.53532814],
            [0.1688554, 0.44463454]]],
     'O': [[0,
            [130.70932, 0.15432897],
            [23.808861, 0.53532814],
            [6.4436083, 0.44463454]],
           [0,
            [5.0331513, -0.09996723],
            [1.1695961, 0.39951283],
            [0.380389, 0.70011547]],
           [1,
            [5.0331513, 0.15591627],
            [1.1695961, 0.60768372],
            [0.380389, 0.39195739]]]}

````{admonition} Extra initialization: Read-input approach
:class: dropdown

`Molecule.charge` and `Molecule.path_dict` should be expicitly defined by caller. If you decided to use PySCF generating integrals, and not using pre-generated integrals in directory `input/h2o/STO-3G`, you are prefered to set this value to `None` or `NotImplemented`. Otherwise, you may explicitly state paths manually as the following toggled code:

````

```python
mole.path_dict = {
    "enuc": "input/h2o/STO-3G/enuc.dat",
    "s": "input/h2o/STO-3G/s.dat",
    "t": "input/h2o/STO-3G/t.dat",
    "v": "input/h2o/STO-3G/v.dat",
    "eri": "input/h2o/STO-3G/eri.dat",
    "mux": "input/h2o/STO-3G/mux.dat",
    "muy": "input/h2o/STO-3G/muy.dat",
    "muz": "input/h2o/STO-3G/muz.dat",
}
```

## Step 1: Nuclear Repulsion Energy

Nuclear Repulsion Energy can be determined as (Szabo, eq 3.185)

$$
E_\mathrm{nuc} = \sum_{B > A} \sum_{A} \frac{Z_A Z_B}{R_{AB}}
$$

````{admonition} PySCF approach
:class: dropdown

PySCF provides a method function that directly calculate nuclear repulsion energy `pyscf.gto.Mole.energy_nuc`.

````

````{admonition} Read-Input approach
:class: dropdown

File {download}`input/h2o/STO-3G/enuc.dat` contains nuclear repulsion energy.

````

````{admonition} Direct approach
:class: dropdown

Actually programming nuclear repulsion energy is not a difficult job. One can simply write two for loops to achieve the task. However, there could be a bonus challenge, only using matrix/tensor operation without for loop, to calculate the energy. The point is

$$
E_\mathrm{nuc} = \frac{1}{2} \sum_{AB} \frac{Z_A Z_B}{R_{AB}}
$$

where we (*unphysically*) define $R_{AB} = +\infty$ if $A = B$.

````

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def eng_nuclear_repulsion(mole: Molecule) -> float:
    # Output: Nuclear Repulsion Energy
    # PySCF       Exactly 1 line of code
    # Read-Input  Exactly 2 or 3 lines of code
    # Direct      About 2~10 lines of code
    raise NotImplementedError("Various approaches may have different lines of code")

Molecule.eng_nuclear_repulsion = eng_nuclear_repulsion
```

### Solution

```python
sol_mole.eng_nuclear_repulsion()
```

    8.00236706181077

## Step 2: One-Electron Integrals

Read various atomic orbital (AO) electron integrals.

**Overlap** `s`

$$
S_{\mu \nu} = \int \phi_\mu (\boldsymbol{r}) \phi_\nu (\boldsymbol{r}) \, \mathrm{d} \boldsymbol{r}
$$

**Kinetic Integral** `t`

$$
T_{\mu \nu} = \int \phi_\mu (\boldsymbol{r}) \left( - \frac{1}{2} \nabla_{\boldsymbol{r}}^2 \right) \phi_\nu (\boldsymbol{r}) \, \mathrm{d} \boldsymbol{r}
$$

**Nuclear-Attraction Integral** `v`

$$
V_{\mu \nu} = \int \phi_\mu (\boldsymbol{r}) \left( - \sum_A \frac{Z_A}{|\boldsymbol{r} - \boldsymbol{R}_A|} \right) \phi_\nu (\boldsymbol{r}) \, \mathrm{d} \boldsymbol{r}
$$

Then form the **core Hamiltonian**:

$$
H_{\mu \nu}^\mathrm{core} = T_{\mu \nu} + V_{\mu \nu}
$$

````{admonition} Hint 1: Proper dimension of matrices
:class: dropdown

It could be important to know how large the matrix is, before we handle electron integrals.

To get the dimension of matrix from fomular, for example overlap $S_{\mu \nu}$, we should take a look at subscripts $\mu \nu$, where it indicates dimension is related to $(\mu, \nu)$. Since $\mu, \nu$ are atomic orbitals, dimension of overlap should be $(n_\mathrm{AO}, n_\mathrm{AO})$.

However, $n_\mathrm{AO}$ can not be apparently got (since it requires summation of basis; however not everyone is required to recite how many basis functions are for Oxygen or Hydrogen DZP, or whether Uraine could be described by STO-3G).

For PySCF approach, one can use `pyscf.gto.Mole.nao` attribute to obtain $n_\mathrm{AO}$. For read-input approach, one could read the last line of file, for example in {download}`input/h2o/STO-3G/t.dat`:
```
    7     7    0.760031883566609
```
Then we should know there are 7 atomic orbitals, i.e. $n_\mathrm{AO} = 7$.

One may implement a short method function `obtain_nao` to store number of atomic orbitals.

````

````{admonition} Hint 2: General function to store 1-electron integral
:class: dropdown

Here we need to read three integrals. Since these three integrals are stored in exactly the same way, so we can make some kind of *abstraction*, i.e. a function which could not only read overlap, but also read kinetic or nuclear attraction integrals.

Recall that all these 1-electron integrals are symmetric matrices (or *permutationally unique*), so in .dat files, only lower-triangle part is stored. However, we prefer to store full matrices in computer memory for coding convenience. Also note that the AO indices on the integrals in the files start with 1 rather than 0.

One may implement a function `read_integral_1e` to read 1-electron integrals.

````

````{admonition} Hint 3: PySCF function pyscf.gto.Mole.intor
:class: dropdown

In PySCF, integrals can be generated by elegent function `pyscf.gto.Mole.intor`. For example, overlap integral can be generated in following code.

```python
>>> sol_mole.mol.intor("int1e_ovlp")
array([[ 1.       ,  0.2367039,  0.       , -0.       ,  0.       ,  0.0384056,  0.0384056],
       [ 0.2367039,  1.       ,  0.       ,  0.       ,  0.       ,  0.3861388,  0.3861388],
       [ 0.       ,  0.       ,  1.       ,  0.       ,  0.       ,  0.2684382, -0.2684382],
       [-0.       ,  0.       ,  0.       ,  1.       ,  0.       ,  0.2097269,  0.2097269],
       [ 0.       ,  0.       ,  0.       ,  0.       ,  1.       ,  0.       ,  0.       ],
       [ 0.0384056,  0.3861388,  0.2684382,  0.2097269,  0.       ,  1.       ,  0.1817599],
       [ 0.0384056,  0.3861388, -0.2684382,  0.2097269,  0.       ,  0.1817599,  1.       ]])
```

Full list of supported integrals are listd in [PySCF API](http://pyscf.org/pyscf/gto.html#pyscf.gto.moleintor.getints). What we need in this step is
- `int1e_ovlp`: overlap
- `int1e_kin`: kinetic integral
- `int1e_nuc`: nuclear-attraction integral

````

### Implementation

You may either want to store the integrals by `Molecule` attributes (in computer memory), or generate integrals on-the-fly (i.e. generate integrals when we aquire them, otherwise they are not stored in computer memory). The inplementation of reference solution uses the latter way.

Reader should fill all `NotImplementedError` in the following code:

```python
def obtain_nao(mole: Molecule, file_path: str=NotImplemented):
    # Input: Any integral file (only used in read-input approach, pyscf approach could simply leave NotImplemented)
    # Attribute Modification: `nao` number of atomic orbitals
    raise NotImplementedError("About 1~3 lines of code")

Molecule.obtain_nao = obtain_nao
```

```python
def read_integral_1e(mole: Molecule, file_path: str) -> np.ndarray:
    # Input: Integral file
    # Output: Symmetric integral matrix
    raise NotImplementedError("About 6~15 lines of code; PySCF approach does not need to implement this function")

Molecule.read_integral_1e = read_integral_1e
```

Reader should choose one approach and remove `NotImplementedError` in the following code:

```python
def integral_ovlp(mole: Molecule) -> np.ndarray:
    # Output: Overlap
    # PySCF        return mole.mol.intor("int1e_ovlp")
    # Read-Input   return mole.read_integral_1e(mole.path_dict["s"])
    raise NotImplementedError("Choose one approach")

def integral_kin(mole: Molecule) -> np.ndarray:
    # Output: Kinetic Integral
    # PySCF        return mole.mol.intor("int1e_kin")
    # Read-Input   return mole.read_integral_1e(mole.path_dict["t"])
    raise NotImplementedError("Choose one approach")

def integral_nuc(mole: Molecule) -> np.ndarray:
    # Output: Nuclear Attraction Integral
    # PySCF        return mole.mol.intor("int1e_nuc")
    # Read-Input   return mole.read_integral_1e(mole.path_dict["v"])
    raise NotImplementedError("Choose one approach")

def get_hcore(mole: Molecule) -> np.ndarray:
    # Output: Hamiltonian Core
    return mole.integral_kin() + mole.integral_nuc()

Molecule.integral_ovlp = integral_ovlp
Molecule.integral_kin = integral_kin
Molecule.integral_nuc = integral_nuc
Molecule.get_hcore = get_hcore
```

### Solution

```python
sol_mole.obtain_nao()
sol_mole.nao
```

    7

```python
sol_mole.get_hcore()
```

    array([[-32.5773954,  -7.5788328,  -0.       ,  -0.0144738,   0.       ,  -1.2401023,  -1.2401023],
           [ -7.5788328,  -9.2009433,  -0.       ,  -0.1768902,   0.       ,  -2.9067098,  -2.9067098],
           [  0.       ,   0.       ,  -7.4588193,   0.       ,   0.       ,  -1.6751501,   1.6751501],
           [ -0.0144738,  -0.1768902,   0.       ,  -7.4153118,   0.       ,  -1.3568683,  -1.3568683],
           [  0.       ,   0.       ,   0.       ,   0.       ,  -7.3471449,   0.       ,   0.       ],
           [ -1.2401023,  -2.9067098,  -1.6751501,  -1.3568683,   0.       ,  -4.5401711,  -1.0711459],
           [ -1.2401023,  -2.9067098,   1.6751501,  -1.3568683,   0.       ,  -1.0711459,  -4.5401711]])

## Step 3: Two-Electron Integrals

Read the two-electron repulsion integrals (electron repulsion integral, ERI) from the {download}`input/h2o/STO-3G/eri.dat` file. The integrals in this file are provided in Mulliken notation over real AO basis functions:

$$
(\mu \nu | \kappa \lambda) = \int \phi_\mu (\boldsymbol{r}_1) \phi_\nu (\boldsymbol{r}_1) \frac{1}{|\boldsymbol{r}_1 - \boldsymbol{r}_2|} \phi_\kappa (\boldsymbol{r}_2) \phi_\lambda (\boldsymbol{r}_2) \, \mathrm{d} \boldsymbol{r}_1 \, \mathrm{d} \boldsymbol{r}_2
$$

Hence, the integrals obey the eight-fold permutational symmetry relationships:

$$
\begin{align}
(\mu \nu | \kappa \lambda) &= (\mu \nu | \lambda \kappa) = (\nu \mu | \kappa \lambda) = (\nu \mu | \lambda \kappa) \\
= (\kappa \lambda | \mu \nu) &= (\lambda \kappa | \mu \nu) = (\kappa \lambda | \nu \mu) = (\lambda \kappa | \nu \mu)
\end{align}
$$

and only the permutationally unique integrals are provided in the file, with the restriction that, for each integral, the following relationships hold:

$$
\mu \geqslant \nu , \quad \kappa \geqslant \lambda \quad \mu \nu \geqslant \kappa \lambda
$$

where

$$
\mu \nu := \mu (\mu + 1) / 2 + \nu \, \quad \kappa \lambda = \kappa (\kappa + 1) / 2 + \lambda
$$

Although two-electron integrals may be stored efficiently in a one-dimensional array and the above relationship, we still use full dimension (i.e. dimension $(\mu, \nu, \kappa, \lambda)$ or $(n_\mathrm{AO}, n_\mathrm{AO}, n_\mathrm{AO}, n_\mathrm{AO})$) to store the two-electron integral tensor, for coding convenience.

If you prefer smaller memory buffer but somehow greater cost of computational efficiency cost and code complexity, original [project 03](https://github.com/CrawfordGroup/ProgrammingProjects/tree/master/Project%2303#step-3-two-electron-integrals) provides some insights on this.

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def read_integral_2e(mole: Molecule, file_path: str) -> np.ndarray:
    # Input: Integral file
    # Output: 8-fold symmetric 2-e integral tensor
    raise NotImplementedError("About 6~20 lines of code; PySCF approach does not need to implement this function")

Molecule.read_integral_2e = read_integral_2e
```

Reader should choose one approach and remove `NotImplementedError` in the following code:

```python
def integral_eri(mole: Molecule) -> np.ndarray:
    # Output: Electron repulsion integral
    # PySCF        return mole.mol.intor("int2e")
    # Read-Input   return mole.read_integral_2e(mole.path_dict["eri"])
    raise NotImplementedError("Choose one approach")

Molecule.integral_eri = integral_eri
```

### Solution

Here we only print one sub-matrix in the electron repulsion integral:

```python
sol_mole.integral_eri()[0, 3]
```

    array([[-0.       , -0.       ,  0.       ,  0.0244774,  0.       ,  0.0006308,  0.0006308],
           [-0.       , -0.       ,  0.       ,  0.0378086,  0.       ,  0.0033729,  0.0033729],
           [ 0.       ,  0.       , -0.       ,  0.       ,  0.       ,  0.0017449, -0.0017449],
           [ 0.0244774,  0.0378086,  0.       , -0.       ,  0.       ,  0.0087746,  0.0087746],
           [ 0.       ,  0.       ,  0.       ,  0.       , -0.       ,  0.       ,  0.       ],
           [ 0.0006308,  0.0033729,  0.0017449,  0.0087746,  0.       ,  0.0063405,  0.0017805],
           [ 0.0006308,  0.0033729, -0.0017449,  0.0087746,  0.       ,  0.0017805,  0.0063405]])

## Step 4: Build the Orthogonalization Matrix

Diagonalize the overlap matrix:

$$
\mathbf{S} \mathbf{L} = \mathbf{L} \mathbf{\Lambda}
$$

where $\mathbf{L}$ is the matrix of eigenvectors (columns) and $\mathbf{\Lambda}$ is the diagonal matrix of corresponding eigenvalues.

Build the *symmetric orthogonalization* matrix using (Szabo and Ostlund, eq 3.167):

$$
\mathbf{S}^{-1/2} = \mathbf{L} \mathbf{\Lambda}^{-1/2} \mathbf{L}^\dagger
$$

where dagger $\dagger$ denotes the matrix transpose.

````{admonition} Scipy approach
:class: dropdown

Actually this process is exactly the same to matrix power (with fractional exponent). This could be done simply by `scipy.linalg.fractional_matrix_power` ([SciPy API](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.fractional_matrix_power.html))

For proper definition of matrix power, see [Matrix function: Diagonalizable matrices](https://en.wikipedia.org/wiki/Matrix_function#Diagonalizable_matrices). Another useful definition could be [Cauchy integral](https://en.wikipedia.org/wiki/Matrix_function#Cauchy_integral) from complex analysis; actually this definition is applied to -1/2 power of matrix in direct random phase approximation (dRPA) correlation energy calculation. Well, in this project, diagonalizable matrix scheme should be sufficient.

````

````{admonition} Direct approach
:class: dropdown

For diagonal matrix, -1/2 power of matrix is equal to -1/2 power to its diagonal values. For example,

$$
\begin{pmatrix} 2 & 0 \\ 0 & 3 \end{pmatrix}^{-1/2} = \begin{pmatrix} 1/\sqrt{2} & 0 \\ 0 & 1/\sqrt{3} \end{pmatrix}
$$

So, not all matrices could be implied with -1/2 power. Only those matrices which eigenvalues are non-zero could be implied with -1/2 power. If negative eigenvalues exists, the matrix implied with -1/2 power could be complex.

````

````{admonition} Note on orthogonalization matrix
:class: dropdown

Our cases in this project is somehow well-behaved. So, the overlap matrices $\mathbf{S}$ would have positive eigenvalues, and orthogonalization matrix is simply symmetric with dimension $(n_\mathrm{AO}, n_\mathrm{AO})$. Thus number of molecular orbital $n_\mathrm{MO}$ is equal to $n_\mathrm{AO}$.

However, when basis become larger, overlap matrices $\mathbf{S}$ could be ill-behaved (linear dependent, or some eigenvalues very close to zero), thus $\mathbf{S}^{-1/2}$ can be numerical instable. Solution to this behavior is discussed in Szabo and Ostlund roughly, around eq 3.172. In this situation, $n_\mathrm{MO} < n_\mathrm{AO}$.

It is important to know that, though in this documentation, $n_\mathrm{MO} = n_\mathrm{AO}$ in most situations, they are essentially two different concepts and does not need to be equal to each other. Molecular orbitals is linear combinition of atomic orbitals (LCAO). Computationally or say in practice, these two values are related by orthogonalization matrix.

````

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def integral_ovlp_m1d2(mole: Molecule) -> np.ndarray:
    # Output: S^{-1/2} in symmetric orthogonalization
    # Scipy       Exactly 1 line of code
    # Direct      About 2~5 lines of code
    raise NotImplementedError("Various approaches may have different lines of code")

Molecule.integral_ovlp_m1d2 = integral_ovlp_m1d2
```

### Solution

```python
sol_mole.integral_ovlp_m1d2()
```

    array([[ 1.0236346, -0.1368547, -0.       , -0.0074873,  0.       ,  0.0190279,  0.0190279],
           [-0.1368547,  1.1578632,  0.       ,  0.0721601, -0.       , -0.2223326, -0.2223326],
           [-0.       ,  0.       ,  1.0733148, -0.       ,  0.       , -0.1757583,  0.1757583],
           [-0.0074873,  0.0721601, -0.       ,  1.038305 , -0.       , -0.1184626, -0.1184626],
           [ 0.       , -0.       ,  0.       , -0.       ,  1.       ,  0.       , -0.       ],
           [ 0.0190279, -0.2223326, -0.1757583, -0.1184626,  0.       ,  1.1297234, -0.0625975],
           [ 0.0190279, -0.2223326,  0.1757583, -0.1184626, -0.       , -0.0625975,  1.1297234]])

## Step 5: Build Fock Matrix

Fock matrix of restricted RHF could be calculated as (Szabo and Ostlund, eq 3.154)

$$
F_{\mu \nu} = H_{\mu \nu}^\mathrm{core} + \sum_{\kappa \lambda} D_{\kappa \lambda} \big[ (\mu \nu | \kappa \lambda) - \frac{1}{2} (\mu \kappa | \nu \lambda) \big]
$$

where the double-summation runs over all the atomic orbitals.

Fock matrix could be regarded as function of density matrix $D_{\kappa \lambda}$. For reproducible and arbitrariness consideration, we may use an arbitary density matrix `dm_ramdom` in the toggled code (notice that density matrix should be symmetry):

```python
np.random.seed(0)
dm_random = np.random.randn(sol_mole.nao, sol_mole.nao)
dm_random += dm_random.T
dm_random
```

    array([[ 3.5281047,  0.2488   ,  1.4226012,  2.8945118,  3.4003372, -0.8209289, -0.7561818],
           [ 0.2488   , -0.2064377,  0.7442728,  1.0084798,  2.9236323,  1.9913284,  2.0724504],
           [ 1.4226012,  0.7442728,  2.9881581, -0.9473233,  0.4680151,  0.3482841, -3.062642 ],
           [ 2.8945118,  1.0084798, -0.9473233,  4.5395092, -1.0762032, -0.3415683, -0.6252582],
           [ 3.4003372,  2.9236323,  0.4680151, -1.0762032, -1.7755715, -2.2830992, -1.6007075],
           [-0.8209289,  1.9913284,  0.3482841, -0.3415683, -2.2830992, -2.0971059, -0.6425276],
           [-0.7561818,  2.0724504, -3.062642 , -0.6252582, -1.6007075, -0.6425276, -3.2277957]])

````{admonition} Hint 1: 4-Dimension tensor contraction: Coulomb
:class: dropdown

There are multiple ways to achieve Coulomb integral

$$
\sum_{\kappa \lambda} D_{\kappa \lambda} (\mu \nu | \kappa \lambda)
$$

One simple way could be for-loop:
```python
eri = mole.integral_eri()
coulomb = np.zeros((mole.nao, mole.nao))
for mu in range(mole.nao):
    for nu in range(mole.nao):
        coulomb[mu, nu] = (eri[mu, nu] * dm_random).sum()
```

Another way is `numpy.einsum` ([NumPy API](https://numpy.org/doc/stable/reference/generated/numpy.einsum.html), [Einstein Summation Convention](https://en.wikipedia.org/wiki/Einstein_notation)):
```python
eri = mole.integral_eri()
coulomb = np.einsum("uvkl, kl -> uv", eri, dm_random)
```

A third way is utilizing numpy boardcasting ([NumPy Docs](https://numpy.org/doc/stable/user/theory.broadcasting.html)):
```python
eri = mole.integral_eri()
coulomb = (eri * dm_random).sum(axis=(-1, -2))
```

Result of the above variable `coulomb` should be
```python
array([[23.9330989,  4.234324 ,  0.0829256,  0.202269 ,  0.3299715,  0.7005698,  0.6955185],
       [ 4.234324 ,  8.9051099,  0.1324784,  0.4010859,  0.9716293,  2.6311948,  2.6251234],
       [ 0.0829256,  0.1324784,  9.0270751, -0.1065442,  0.0349767,  1.596199 , -1.5653815],
       [ 0.202269 ,  0.4010859, -0.1065442,  9.0874751, -0.1440637,  1.2958253,  1.3189783],
       [ 0.3299715,  0.9716293,  0.0349767, -0.1440637,  8.5657791,  0.2010907,  0.1926869],
       [ 0.7005698,  2.6311948,  1.596199 ,  1.2958253,  0.2010907,  3.7626551,  0.8928712],
       [ 0.6955185,  2.6251234, -1.5653815,  1.3189783,  0.1926869,  0.8928712,  3.7152703]])
```

````

````{admonition} Hint 2: 4-Dimension tensor contraction: Exchange
:class: dropdown

There are also multiple ways to achieve Exchange integral

$$
\sum_{\kappa \lambda} D_{\kappa \lambda} (\mu \kappa | \nu \lambda)
$$

One simple way could be for-loop:
```python
eri = mole.integral_eri()
exchange = np.zeros((mole.nao, mole.nao))
for mu in range(mole.nao):
    for nu in range(mole.nao):
        exchange[mu, nu] = (eri[mu, :, nu] * dm_random).sum()
```

Another way is `numpy.einsum` ([NumPy API](https://numpy.org/doc/stable/reference/generated/numpy.einsum.html), [Einstein Summation Convention](https://en.wikipedia.org/wiki/Einstein_notation)):
```python
eri = mole.integral_eri()
exchange = np.einsum("ukvl, kl -> uv", eri, dm_random)
```

A third way is utilizing numpy boardcasting ([NumPy Docs](https://numpy.org/doc/stable/user/theory.broadcasting.html)):
```python
eri = mole.integral_eri()
exchange = (eri * dm_random[:, None, :]).sum(axis=(-1, -3))
```

Result of the above variable `exchange` should be
```python
array([[17.1408607,  2.9642433,  1.6954651,  3.4316066,  4.5482841,  1.0611853,  0.5458695],
       [ 2.9642433,  2.8747645,  0.352244 ,  1.8035899,  2.7594346,  1.3116692,  1.3224965],
       [ 1.6954651,  0.352244 ,  4.1142193, -1.0576397,  0.2781629,  0.6138329, -1.6697351],
       [ 3.4316066,  1.8035899, -1.0576397,  4.0318516, -1.39224  ,  0.5225643,  0.8203703],
       [ 4.5482841,  2.7594346,  0.2781629, -1.39224  , -0.8581923, -0.220426 , -0.137229 ],
       [ 1.0611853,  1.3116692,  0.6138329,  0.5225643, -0.220426 , -0.2555552,  0.0139439],
       [ 0.5458695,  1.3224965, -1.6697351,  0.8203703, -0.137229 ,  0.0139439,  0.006697 ]])
```

````

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def get_fock(mole: Molecule, dm: np.ndarray) -> np.ndarray:
    # Input: Density matrix
    # Output: Fock matrix (based on input density matrix, not only final fock matrix)
    raise NotImplementedError("About 1~10 lines of code")

Molecule.get_fock = get_fock
```

### Solution

```python
sol_mole.get_fock(dm_random)
```

    array([[-17.2147269,  -4.8266305,  -0.7648069,  -1.5280082,  -1.9441706,  -1.0701252,  -0.8175185],
           [ -4.8266305,  -1.7332157,  -0.0436436,  -0.6775993,  -0.408088 ,  -0.9313496,  -0.9428347],
           [ -0.7648069,  -0.0436436,  -0.4888538,   0.4222757,  -0.1041047,  -0.3858676,   0.9446363],
           [ -1.5280082,  -0.6775993,   0.4222757,  -0.3437624,   0.5520563,  -0.3223251,  -0.4480752],
           [ -1.9441706,  -0.408088 ,  -0.1041047,   0.5520563,   1.6477304,   0.3113038,   0.2613014],
           [ -1.0701252,  -0.9313496,  -0.3858676,  -0.3223251,   0.3113038,  -0.6497384,  -0.1852467],
           [ -0.8175185,  -0.9428347,   0.9446363,  -0.4480752,   0.2613014,  -0.1852467,  -0.8282493]])

## Step 6: Orbital Coefficients

Orbital coefficients $C_{\mu p}$ can be obtained from Fock matrix diagonalization (Roothaan equation, Szabo and Ostlund, eq 3.138-139):

$$
\mathbf{F} \mathbf{C} = \mathbf{S} \mathbf{C} \boldsymbol{\varepsilon}
$$

or in element of matrix form:

$$
\sum_\nu F_{\mu \nu} C_{\nu i} = \varepsilon_i \sum_v S_{\mu \nu} C_{\nu i}
$$

````{admonition} Conventional approach
:class: dropdown

The conventional approach is described in Szabo and Ostlund, eq 3.174 - 3.178. First, we need to obtain transformed Fock matrix $\mathbf{F}'$ :

$$
\mathbf{F}' = \mathbf{S}^{-1/2} \mathbf{F} \mathbf{S}^{-1/2}
$$

Then obtain transformed orbital coefficient $\mathbf{C}'$ from the following transformed Roothaan equation:

$$
\mathbf{F}' \mathbf{C}' = \mathbf{C}' \boldsymbol{\varepsilon}
$$

Then we may get the orbital coefficient $\mathbf{C}$ by:

$$
\mathbf{C} = \mathbf{S}^{-1/2} \mathbf{C}'
$$

````

````{admonition} SciPy approach
:class: dropdown

Actually, SciPy can directly solve Roothaan equation, i.e. $\mathbf{F} \mathbf{C} = \mathbf{S} \mathbf{C} \boldsymbol{\varepsilon}$. See `scipy.linalg.eigh` ([SciPy API](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.eigh.html)).

Although this function is very similar to `numpy.linalg.eigh`, `scipy.linalg.eigh` can be more powerful.

````

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def get_coeff_from_fock_diag(mole: Molecule, fock: np.ndarray) -> np.ndarray:
    # Input: Fock matrix
    # Output: Orbital coefficient obtained by fock matrix diagonalization
    raise NotImplementedError("About 1~10 lines of code")

Molecule.get_coeff_from_fock_diag = get_coeff_from_fock_diag
```

### Solution

We may use the Fock matrix `fock` generated in Step 5 to test whether our function works correctly.

```python
fock = sol_mole.get_fock(dm_random)
sol_mole.get_coeff_from_fock_diag(fock)
```

    array([[-0.9758933,  0.0871776, -0.1396227,  0.076924 ,  0.2567259,  0.080758 , -0.0976095],
           [-0.0642153, -0.2464221,  0.3462111, -0.3903606, -0.9492593, -0.4619375, -0.1089793],
           [-0.0450585,  0.4811949,  0.2583564, -0.1081134, -0.4068632,  0.8556978,  0.0633632],
           [-0.0936041, -0.3274823,  0.081244 ,  0.8368082, -0.4809293,  0.014067 ,  0.2388754],
           [-0.0980952,  0.1389688, -0.1311965, -0.2212164, -0.0206398, -0.1509956,  0.9389836],
           [ 0.028227 , -0.0055787,  0.7238113,  0.0903444,  0.8738889, -0.2600127,  0.1035881],
           [ 0.0186318, -0.5265492, -0.1712072, -0.377253 ,  0.3074832,  0.9038413,  0.1191796]])

## Step 7: Build Density Matrix

Build the (restricted) density matrix using the occupied molecular orbital coefficients (Szabo and Ostlund, eq 3.145):

$$
D_{\mu \nu} = 2 \sum_i C_{\mu i} C_{\nu i}
$$

````{admonition} Hint 1: Summation range
:class: dropdown

Notice that not all orbital coefficient are taken into account. Only occupied molecular orbital coefficients is included.

In the figure above, matrix in the middle is the usual representation of orbital coefficients $C_{\mu p}$, where row denotes atomic orbital subscript $\mu$, and column denotes molecular orbital subscript $\nu$. For water molecule, there are 10 electrons, so number of occupied orbitals is 10/2 = 5. So only the first 5 columns (area painted in blue) is considered as $C_{\mu i}$.

We don't have a function determining number of occupied orbitals so far. So the reader is encouraged to implement a simple function to do this job.

````

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def obtain_nocc(mole: Molecule):
    # Attribute Modification: `nocc` occupied orbital
    raise NotImplementedError("About 1~5 lines of code")

Molecule.obtain_nocc = obtain_nocc
```

```python
def make_rdm1(mole: Molecule, coeff: np.ndarray) -> np.ndarray:
    # Input: molecular orbital coefficient
    # Output: density for the given orbital coefficient
    raise NotImplementedError("About 1~5 lines of code")

Molecule.make_rdm1 = make_rdm1
```

### Solution

We may use the Fock matrix `coeff` generated in Step 6 to test whether our function works correctly.

```python
sol_mole.obtain_nocc()
coeff = sol_mole.get_coeff_from_fock_diag(sol_mole.get_fock(dm_random))
sol_mole.make_rdm1(coeff)
```

    array([[ 2.1025753, -0.5617635, -0.1258392, -0.0152828,  0.2076956,  0.2044125,  0.0194752],
           [-0.5617635,  2.4763685,  0.8043683,  0.489414 ,  0.0651585, -1.229321 , -0.1506671],
           [-0.1258392,  0.8043683,  0.955106 , -0.0543459,  0.1394193, -0.3645514, -0.7655245],
           [-0.0152828,  0.489414 , -0.0543459,  2.1082959, -0.444352 , -0.5733756, -0.6135682],
           [ 0.2076956,  0.0651585,  0.1394193, -0.444352 ,  0.1910204, -0.2730566,  0.0491367],
           [ 0.2044125, -1.229321 , -0.3645514, -0.5733756, -0.2730566,  2.5931491,  0.2283302],
           [ 0.0194752, -0.1506671, -0.7655245, -0.6135682,  0.0491367,  0.2283302,  1.0875576]])

## Step 8: Update Density Matrix

Review procedure from Step 5 to Step 7:
- Step 5: $D_{\mu \nu} \rightarrow F_{\mu \nu}$
- Step 6: $F_{\mu \nu} \rightarrow C_{\mu p}$
- Step 7: $C_{\mu p} \rightarrow D_{\mu \nu}$

Run these processes in series, and the density updated. If we just simply loop these three steps infinitely, then we except the density converged to stable, i.e. self-consistent.

Write a function that inputs a guess density matrix, output an updated density matrix.

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def get_updated_dm(mole: Molecule, dm: np.ndarray) -> np.ndarray:
    # Input: Guess density matrix
    # Output: Updated density matrix
    raise NotImplementedError("About 1~5 lines of code")

Molecule.get_updated_dm = get_updated_dm
```

### Solution

We use the random-generated density matrix `dm_random` in Step 5 to test whether our function works correctly.

```python
sol_mole.get_updated_dm(dm_random)
```

    array([[ 2.1025753, -0.5617635, -0.1258392, -0.0152828,  0.2076956,  0.2044125,  0.0194752],
           [-0.5617635,  2.4763685,  0.8043683,  0.489414 ,  0.0651585, -1.229321 , -0.1506671],
           [-0.1258392,  0.8043683,  0.955106 , -0.0543459,  0.1394193, -0.3645514, -0.7655245],
           [-0.0152828,  0.489414 , -0.0543459,  2.1082959, -0.444352 , -0.5733756, -0.6135682],
           [ 0.2076956,  0.0651585,  0.1394193, -0.444352 ,  0.1910204, -0.2730566,  0.0491367],
           [ 0.2044125, -1.229321 , -0.3645514, -0.5733756, -0.2730566,  2.5931491,  0.2283302],
           [ 0.0194752, -0.1506671, -0.7655245, -0.6135682,  0.0491367,  0.2283302,  1.0875576]])

## Step 9: Compute the Total Energy

Compute the total energy as (Szabo and Ostlund, eq 3.184 - 3.185)

$$
E_\mathrm{tot} = \frac{1}{2} \sum_{\mu \nu} D_{\mu \nu} (H_{\mu \nu}^\mathrm{core} + F_{\mu \nu}) + E_\mathrm{nuc}
$$

where $F_{\mu \nu}$ is obtained from the the same density matrix $D_{\mu \nu}$ in the fomular above. Indeed, $E_\mathrm{tot}$ could be regarded as a function of density matrix $D_{\mu \nu}$. $E_\mathrm{nuc}$ has been already obtained in Step 1.

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def eng_total(mole: Molecule, dm: np.ndarray) -> float:
    # Input: Any density matrix (although one should impose tr(D@S)=nelec to satisfy variational principle)
    # Output: SCF energy (electronic contribution + nuclear repulsion contribution)
    raise NotImplementedError("About 1~5 lines of code")

Molecule.eng_total = eng_total
```

### Solution

We use the random-generated density matrix `dm_random` in Step 5 to test whether our function works correctly.

```python
sol_mole.eng_total(dm_random)
```

    -126.934270832249

````{warning}

You may find the converged energy is about -75 Hartree (a.u.). Maybe to you, it is very suspicious that some density could yield much lower energy (-127) than the variational minimum (-75). This is because the density `dm_random` does not satisfy boundary condition $\mathrm{tr} (\mathbf{DS}) = n_\mathrm{elec}$.

````

## Step 10: SCF Loop and Convergence

Given initial (empty) density $D_{\mu \nu}^0 = 0$ and initial energy value $E_\mathrm{tot}^0 = 0$. Use the function described in Step 9 to update density matrix until convergence.

What defines convergence is energy and density criterion:

$$
\begin{align}
\Delta_E &= |E_\mathrm{tot}^t - E_\mathrm{tot}^{t-1}| < \delta_1 \\
\mathrm{rms}_\mathbf{D} &= \left[ \sum_{\mu \nu} (D_{\mu \nu}^t - D_{\mu \nu}^{t-1})^2 \right]^{-1/2} < \delta_2
\end{align}
$$

where superscript $t$ means self-consistent field loop iteration. $\mathrm{rms}_\mathbf{D}$ is actually Frobenius norm of the difference in consecutive density matrix.

In our implementation, we set criterion

$$
\delta_1 = 10^{-10} \, \mathsf{a.u.}, \quad \delta_2 = 10^{-8} \, \mathsf{a.u.}
$$

and maximum iteration number set to 64.

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def scf_process(mole: Molecule, dm_guess: np.ndarray=None) -> Tuple[float, np.ndarray]:
    # Input: Density matrix guess
    # Output: Converged SCF total energy and density matrix
    eng, dm = 0., np.zeros((mole.nao, mole.nao)) if dm_guess is None else np.copy(dm_guess)
    eng_next, dm_next = NotImplemented, NotImplemented
    max_iter, thresh_eng, thresh_dm = 64, 1e-10, 1e-8
    print("{:>5} {:>20} {:>20} {:>20}".format("Epoch", "Total Energy", "Energy Deviation", "Density Deviation"))
    for epoch in range(max_iter):
        raise NotImplemented("Calculate total energy and update density. About 2 lines of code")
        print("{:5d} {:20.12f} {:20.12f} {:20.12f}".format(epoch, eng_next, eng_next - eng, np.linalg.norm(dm_next - dm)))
        raise NotImplemented("Test convergence criterion and judge if we should break. About 2 lines of code")
        eng, dm = eng_next, dm_next
    return eng, dm
```

### Solution

```python
dm_guess = np.zeros((sol_mole.nao, sol_mole.nao))
eng_converged, dm_converged = sol_mole.scf_process(dm_guess)
eng_converged, dm_converged
```

    Epoch         Total Energy     Energy Deviation    Density Deviation
        0       8.002367061811       8.002367061811       5.100522155128
        1     -73.285796421100     -81.288163482911       3.653346168960
        2     -74.828125379745      -1.542328958644       0.959140729724
        3     -74.935487998012      -0.107362618267       0.173663377814
        4     -74.941477742494      -0.005989744483       0.062052272719
        5     -74.941971996116      -0.000494253622       0.021598566359
        6     -74.942056061414      -0.000084065298       0.010509653663
        7     -74.942074419139      -0.000018357725       0.004877159285
        8     -74.942078647640      -0.000004228501       0.002354559061
        9     -74.942079630140      -0.000000982500       0.001129086359
       10     -74.942079858804      -0.000000228664       0.000544409983
       11     -74.942079912038      -0.000000053234       0.000262360618
       12     -74.942079924431      -0.000000012394       0.000126551333
       13     -74.942079927317      -0.000000002885       0.000061045094
       14     -74.942079927988      -0.000000000672       0.000029451693
       15     -74.942079928145      -0.000000000156       0.000014209673
       16     -74.942079928181      -0.000000000036       0.000006856052
       17     -74.942079928190      -0.000000000008       0.000003308028
       18     -74.942079928192      -0.000000000002       0.000001596130
       19     -74.942079928192      -0.000000000000       0.000000770138
       20     -74.942079928192      -0.000000000000       0.000000371595
       21     -74.942079928192      -0.000000000000       0.000000179297
       22     -74.942079928192       0.000000000000       0.000000086512
       23     -74.942079928192      -0.000000000000       0.000000041742
       24     -74.942079928192       0.000000000000       0.000000020141
       25     -74.942079928192      -0.000000000000       0.000000009718

    (-74.94207992819236,
     array([[ 2.1097473, -0.4655955, -0.       ,  0.1052458,  0.       , -0.0163012, -0.0163012],
            [-0.4655955,  2.0302217, -0.       , -0.5646001, -0.       , -0.0610789, -0.0610789],
            [-0.       , -0.       ,  0.7375898,  0.       , -0.       ,  0.5501973, -0.5501973],
            [ 0.1052458, -0.5646001,  0.       ,  1.1320837,  0.       ,  0.51723  ,  0.51723  ],
            [ 0.       , -0.       , -0.       ,  0.       ,  2.       ,  0.       , -0.       ],
            [-0.0163012, -0.0610789,  0.5501973,  0.51723  ,  0.       ,  0.6690534, -0.1517744],
            [-0.0163012, -0.0610789, -0.5501973,  0.51723  , -0.       , -0.1517744,  0.6690534]]))

## Addition 1: Dipole Moment

As discussed in detail in Ch. 3 of the text by Szabo and Ostlund, the calculation of one-electron properties requires density matrix and the relevant property integrals. The electronic contribution to the electric-dipole moment may be computed using (Szabo and Ostlund, eq 3.190 - 3.192)

$$
\boldsymbol{\mu} = - \sum_{\mu \nu} P_{\mu \nu} (\mu | \boldsymbol{r} | \nu) + \sum_A Z_A \boldsymbol{R}_A
$$

````{admonition} Hint 1: Access to integrals
:class: dropdown

Integral $(\mu | \boldsymbol{r} | \nu)$ is stored in $\boldsymbol{r}$'s three components $(x, y, z)$. $x$ component is stored in file {download}`input/h2o/STO-3G/mux.dat`. For PySCF users, simply feed `"int1e_r"` to `pyscf.gto.Mole.intor` and integral $(\mu | \boldsymbol{r} | \nu)$ is returned in dimension $(3, n_\mathrm{AO}, n_\mathrm{AO})$.

````

### Implementation

Reader should fill all `NotImplementedError` in the following code:

```python
def integral_dipole(mole: Molecule) -> np.ndarray:
    # Output: Dipole related integral (mu|r|nu) in dimension (3, nao, nao)
    raise NotImplementedError("About 1~5 lines of code")

Molecule.integral_dipole = integral_dipole
```

```python
def get_dipole(mole: Molecule, dm: np.array) -> np.ndarray:
    # Input: SCF converged density matrix
    # Output: Molecule dipole in A.U.
    raise NotImplementedError("About 1~5 lines of code")

Molecule.get_dipole = get_dipole
```

### Solution

```python
sol_mole.integral_dipole()
```

    array([[[ 0.       ,  0.       ,  0.0507919,  0.       ,  0.       ,  0.0022297, -0.0022297],
            [ 0.       ,  0.       ,  0.6411728,  0.       ,  0.       ,  0.2627425, -0.2627425],
            [ 0.0507919,  0.6411728,  0.       ,  0.       ,  0.       ,  0.4376306,  0.4376306],
            [ 0.       ,  0.       ,  0.       ,  0.       ,  0.       ,  0.1473994, -0.1473994],
            [ 0.       ,  0.       ,  0.       ,  0.       ,  0.       ,  0.       ,  0.       ],
            [ 0.0022297,  0.2627425,  0.4376306,  0.1473994,  0.       ,  1.6380368, -0.       ],
            [-0.0022297, -0.2627425,  0.4376306, -0.1473994,  0.       ,  0.       , -1.6380368]],

           [[-0.1432258, -0.0339021,  0.       ,  0.0507919,  0.       , -0.0037587, -0.0037587],
            [-0.0339021, -0.1432258,  0.       ,  0.6411728,  0.       ,  0.1499719,  0.1499719],
            [ 0.       ,  0.       , -0.1432258,  0.       ,  0.       ,  0.1089522, -0.1089522],
            [ 0.0507919,  0.6411728,  0.       , -0.1432258,  0.       ,  0.3340907,  0.3340907],
            [ 0.       ,  0.       ,  0.       ,  0.       , -0.1432258,  0.       ,  0.       ],
            [-0.0037587,  0.1499719,  0.1089522,  0.3340907,  0.       ,  1.1365488,  0.206579 ],
            [-0.0037587,  0.1499719, -0.1089522,  0.3340907,  0.       ,  0.206579 ,  1.1365488]],

           [[ 0.       ,  0.       ,  0.       ,  0.       ,  0.0507919,  0.       ,  0.       ],
            [ 0.       ,  0.       ,  0.       ,  0.       ,  0.6411728,  0.       ,  0.       ],
            [ 0.       ,  0.       ,  0.       ,  0.       ,  0.       ,  0.       ,  0.       ],
            [ 0.       ,  0.       ,  0.       ,  0.       ,  0.       ,  0.       ,  0.       ],
            [ 0.0507919,  0.6411728,  0.       ,  0.       ,  0.       ,  0.248968 ,  0.248968 ],
            [ 0.       ,  0.       ,  0.       ,  0.       ,  0.248968 ,  0.       ,  0.       ],
            [ 0.       ,  0.       ,  0.       ,  0.       ,  0.248968 ,  0.       ,  0.       ]]])

```python
sol_mole.get_dipole(dm_converged)
```

    array([0.       , 0.6035213, 0.       ])

## Addition 2: Mulliken Population Analysis

A Mulliken population analysis (also described in Szabo and Ostlund, Ch. 3) requires the overlap integrals and the electron density, in addition to information about the number of basis functions centered on each atom. The charge on atom A may be computed as (Szabo and Ostlund, eq 3.196)

$$
q_A = Z_A - \sum_{\mu \in A} \sum_\nu P_{\mu \nu} S_{\nu \mu}
$$

where the summation is limited to only those basis functions centered on atom $A$.

````{admonition} PySCF Approach
:class: dropdown

Due to limited file input, this task could only be accomplished by the help from PySCF. Function `pyscf.gto.Mole.aoslice_by_atom` could help group atomic orbitals by its atom center.

````

### Implementation

This task is a little difficult, due to $\mu \in A$ is hard to figure out by program. So this task is not required to be accomplished by one's own. However, hard-coding is possible for a specified system, for example water/STO-3G.

Reader may try to fill `NotImplementedError` in the following code:

```python
def population_analysis(mole: Molecule, dm: np.array) -> np.ndarray:
    # Input: SCF converged density matrix
    # Output: Mulliken population analysis, charges on every atom
    raise NotImplementedError("About 5~10 lines of code")

Molecule.population_analysis = population_analysis
```

### Solution

```python
sol_mole.population_analysis(dm_converged)
```

    array([-0.2531461,  0.126573 ,  0.126573 ])

## Test Cases

**Water/STO-3G** ([Directory](https://github.com/ajz34/PyCrawfordProgProj/tree/master/source/Project_03/input/h2o/STO-3G))

```python
sol_mole = SolMol()
sol_mole.construct_from_dat_file("input/h2o/STO-3G/geom.dat")
sol_mole.obtain_mol_instance(basis="STO-3G")
sol_mole.obtain_nocc()
sol_mole.path_dict = {
    "enuc": "input/h2o/STO-3G/enuc.dat",
    "s":    "input/h2o/STO-3G/s.dat",
    "t":    "input/h2o/STO-3G/t.dat",
    "v":    "input/h2o/STO-3G/v.dat",
    "eri":  "input/h2o/STO-3G/eri.dat",
    "mux":  "input/h2o/STO-3G/mux.dat",
    "muy":  "input/h2o/STO-3G/muy.dat",
    "muz":  "input/h2o/STO-3G/muz.dat",
}
sol_mole.obtain_nao(file_path=sol_mole.path_dict["s"])
sol_mole.print_solution_03()
```

    === SCF Process ===
    Epoch         Total Energy     Energy Deviation    Density Deviation
        0       8.002367061811       8.002367061811       5.100522155128
        1     -73.285796421100     -81.288163482911       3.653346168960
        2     -74.828125379745      -1.542328958644       0.959140729724
        3     -74.935487998012      -0.107362618267       0.173663377814
        4     -74.941477742494      -0.005989744483       0.062052272719
        5     -74.941971996116      -0.000494253622       0.021598566359
        6     -74.942056061414      -0.000084065298       0.010509653663
        7     -74.942074419139      -0.000018357725       0.004877159285
        8     -74.942078647640      -0.000004228501       0.002354559061
        9     -74.942079630140      -0.000000982500       0.001129086359
       10     -74.942079858804      -0.000000228664       0.000544409983
       11     -74.942079912038      -0.000000053234       0.000262360618
       12     -74.942079924431      -0.000000012394       0.000126551333
       13     -74.942079927317      -0.000000002885       0.000061045094
       14     -74.942079927988      -0.000000000672       0.000029451693
       15     -74.942079928145      -0.000000000156       0.000014209673
       16     -74.942079928181      -0.000000000036       0.000006856052
       17     -74.942079928190      -0.000000000008       0.000003308028
       18     -74.942079928192      -0.000000000002       0.000001596130
       19     -74.942079928192      -0.000000000000       0.000000770138
       20     -74.942079928192      -0.000000000000       0.000000371595
       21     -74.942079928192      -0.000000000000       0.000000179297
       22     -74.942079928192       0.000000000000       0.000000086512
       23     -74.942079928192      -0.000000000000       0.000000041742
       24     -74.942079928192       0.000000000000       0.000000020141
       25     -74.942079928192      -0.000000000000       0.000000009718
    === Dipole (in A.U.) ===
    [0.        0.6035213 0.       ]
    === Mulliken Population Analysis ===
    [-0.2531461  0.126573   0.126573 ]

**Water/DZ** ([Directory](https://github.com/ajz34/PyCrawfordProgProj/tree/master/source/Project_03/input/h2o/DZ))

```python
sol_mole = SolMol()
sol_mole.construct_from_dat_file("input/h2o/DZ/geom.dat")
sol_mole.obtain_mol_instance(basis="DZ")
sol_mole.obtain_nocc()
sol_mole.path_dict = {
    "enuc": "input/h2o/DZ/enuc.dat",
    "s":    "input/h2o/DZ/s.dat",
    "t":    "input/h2o/DZ/t.dat",
    "v":    "input/h2o/DZ/v.dat",
    "eri":  "input/h2o/DZ/eri.dat",
    "mux":  "input/h2o/DZ/mux.dat",
    "muy":  "input/h2o/DZ/muy.dat",
    "muz":  "input/h2o/DZ/muz.dat",
}
sol_mole.obtain_nao(file_path=sol_mole.path_dict["s"])
# sol_mole.print_solution_03()
sol_mole.mol._basis
```

    {'H': [[0, [19.2406, 0.032828], [2.8992, 0.231208], [0.6534, 0.817238]],
      [0, [0.1776, 1.0]]],
     'O': [[0,
       [7816.54, 0.002031],
       [1175.82, 0.015436],
       [273.188, 0.073771],
       [81.1696, 0.247606],
       [27.1836, 0.611832],
       [3.4136, 0.241205]],
      [0, [9.5322, 1.0]],
      [0, [0.9398, 1.0]],
      [0, [0.2846, 1.0]],
      [1,
       [35.1832, 0.01958],
       [7.904, 0.124189],
       [2.3051, 0.394727],
       [0.7171, 0.627375]],
      [1, [0.2137, 1.0]]]}

**Water/DZP** ([Directory](https://github.com/ajz34/PyCrawfordProgProj/tree/master/source/Project_03/input/h2o/DZP)) For this molecule, read-input approach is prefered. For explanation, see comment in code.

```python
sol_mole = SolMol()
sol_mole.construct_from_dat_file("input/h2o/DZP/geom.dat")
sol_mole.obtain_mol_instance(basis="DZP_Dunning")
# The following 3 lines of code is only used to reproduce the exact integral values
# That is using cartesian orbital (d shell), and one s orbital of H atom become more diffuse
# Guess the original Crawford project is psi3-era, since this basis is psi3-dzp.gbs in psi4
# The reader could just take these 3 lines without any consideration now
sol_mole.mol._basis["H"][2][1][0] = 0.75
sol_mole.mol.cart = True
sol_mole.mol.build()
sol_mole.obtain_nocc()
sol_mole.path_dict = {
    "enuc": "input/h2o/DZP/enuc.dat",
    "s":    "input/h2o/DZP/s.dat",
    "t":    "input/h2o/DZP/t.dat",
    "v":    "input/h2o/DZP/v.dat",
    "eri":  "input/h2o/DZP/eri.dat",
    "mux":  "input/h2o/DZP/mux.dat",
    "muy":  "input/h2o/DZP/muy.dat",
    "muz":  "input/h2o/DZP/muz.dat",
}
sol_mole.obtain_nao(file_path=sol_mole.path_dict["s"])
sol_mole.print_solution_03()
```

    === SCF Process ===
    Epoch         Total Energy     Energy Deviation    Density Deviation
        0       8.002367061811       8.002367061811       6.257265233492
        1     -69.170053861694     -77.172420923505      17.801143659997
        2     -71.415303617369      -2.245249755675      17.110557111597
        3     -73.721678035341      -2.306374417971       5.307965471502
        4     -74.381838191510      -0.660160156170       4.571348737722
        5     -75.161436131461      -0.779597939951       2.439102528531
        6     -75.510928746774      -0.349492615314       1.930041242703
        7     -75.762675155351      -0.251746408577       1.232205113151
        8     -75.879680085321      -0.117004929970       0.916128789513
        9     -75.946361719726      -0.066681634405       0.613780446754
       10     -75.977561890457      -0.031200170731       0.442401057269
       11     -75.993744120619      -0.016182230162       0.301602891297
       12     -76.001410814858      -0.007666694239       0.213952051504
       13     -76.005239911770      -0.003829096912       0.147280173556
       14     -76.007073420274      -0.001833508504       0.103604143871
       15     -76.007975034687      -0.000901614413       0.071717537053
       16     -76.008409647227      -0.000434612540       0.050226193234
       17     -76.008621929669      -0.000212282442       0.034875554059
       18     -76.008724632743      -0.000102703074       0.024367318109
       19     -76.008774643915      -0.000050011172       0.016948253356
       20     -76.008798885129      -0.000024241215       0.011826975181
       21     -76.008810672478      -0.000011787348       0.008233394702
       22     -76.008816391373      -0.000005718895       0.005741738436
       23     -76.008819170302      -0.000002778929       0.003999031169
       24     -76.008820519187      -0.000001348884       0.002787846838
       25     -76.008821174422      -0.000000655235       0.001942180465
       26     -76.008821492544      -0.000000318122       0.001353705229
       27     -76.008821647050      -0.000000154506       0.000943197655
       28     -76.008821722072      -0.000000075022       0.000657347543
       29     -76.008821758507      -0.000000036434       0.000458041038
       30     -76.008821776199      -0.000000017692       0.000319208407
       31     -76.008821784790      -0.000000008592       0.000222433401
       32     -76.008821788962      -0.000000004172       0.000155009394
       33     -76.008821790989      -0.000000002026       0.000108017071
       34     -76.008821791972      -0.000000000984       0.000075273839
       35     -76.008821792450      -0.000000000478       0.000052454539
       36     -76.008821792682      -0.000000000232       0.000036553701
       37     -76.008821792795      -0.000000000113       0.000025472575
       38     -76.008821792850      -0.000000000055       0.000017750857
       39     -76.008821792876      -0.000000000027       0.000012369785
       40     -76.008821792889      -0.000000000013       0.000008620007
       41     -76.008821792895      -0.000000000006       0.000006006911
       42     -76.008821792898      -0.000000000003       0.000004185970
       43     -76.008821792900      -0.000000000001       0.000002917024
       44     -76.008821792901      -0.000000000001       0.000002032754
       45     -76.008821792901      -0.000000000000       0.000001416540
       46     -76.008821792901      -0.000000000000       0.000000987128
       47     -76.008821792901       0.000000000000       0.000000687888
       48     -76.008821792901      -0.000000000000       0.000000479360
       49     -76.008821792901       0.000000000000       0.000000334046
       50     -76.008821792901      -0.000000000000       0.000000232783
       51     -76.008821792901      -0.000000000000       0.000000162217
       52     -76.008821792901       0.000000000000       0.000000113042
       53     -76.008821792901       0.000000000000       0.000000078774
       54     -76.008821792901      -0.000000000000       0.000000054894
       55     -76.008821792901       0.000000000000       0.000000038254
       56     -76.008821792901      -0.000000000000       0.000000026657
       57     -76.008821792901      -0.000000000000       0.000000018576
       58     -76.008821792901       0.000000000000       0.000000012945
       59     -76.008821792901      -0.000000000000       0.000000009021
    === Dipole (in A.U.) ===
    [-0.         0.9026624 -0.       ]
    === Mulliken Population Analysis ===
    [-0.6109947  0.3054973  0.3054973]

**Methane/STO-3G** ([Directory](https://github.com/ajz34/PyCrawfordProgProj/tree/master/source/Project_03/input/ch4/STO-3G))

```python
sol_mole = SolMol()
sol_mole.construct_from_dat_file("input/ch4/STO-3G/geom.dat")
sol_mole.obtain_mol_instance(basis="STO-3G")
sol_mole.obtain_nocc()
sol_mole.path_dict = {
    "enuc": "input/ch4/STO-3G/enuc.dat",
    "s":    "input/ch4/STO-3G/s.dat",
    "t":    "input/ch4/STO-3G/t.dat",
    "v":    "input/ch4/STO-3G/v.dat",
    "eri":  "input/ch4/STO-3G/eri.dat",
    "mux":  "input/ch4/STO-3G/mux.dat",
    "muy":  "input/ch4/STO-3G/muy.dat",
    "muz":  "input/ch4/STO-3G/muz.dat",
}
sol_mole.obtain_nao(file_path=sol_mole.path_dict["s"])
sol_mole.print_solution_03()
```

    === SCF Process ===
    Epoch         Total Energy     Energy Deviation    Density Deviation
        0      13.497304462033      13.497304462033       6.515740378275
        1     -36.083448573242     -49.580753035275       6.436541620173
        2     -39.564513419144      -3.481064845902       1.104568450885
        3     -39.721836323727      -0.157322904583       0.195127470881
        4     -39.726692997946      -0.004856674219       0.034000759430
        5     -39.726845353413      -0.000152355467       0.006127229495
        6     -39.726850157495      -0.000004804082       0.001066349141
        7     -39.726850311142      -0.000000153647       0.000197843234
        8     -39.726850316180      -0.000000005038       0.000034671446
        9     -39.726850316352      -0.000000000173       0.000006876790
       10     -39.726850316359      -0.000000000006       0.000001267090
       11     -39.726850316359      -0.000000000000       0.000000281235
       12     -39.726850316359       0.000000000000       0.000000057836
       13     -39.726850316359       0.000000000000       0.000000014097
       14     -39.726850316359      -0.000000000000       0.000000003204
    === Dipole (in A.U.) ===
    [-0.  0.  0.]
    === Mulliken Population Analysis ===
    [-0.2604309  0.0651077  0.0651077  0.0651077  0.0651077]

## References

- Szabo, A.; Ostlund, N. S. *Modern Quantum Chemistry: Introduction to Advanced Electronic Structure Theory* Dover Publication Inc., 1996.

    ISBN-13: 978-0486691862


# Linear Regression Example Notebook
This example shows how to predict house prices using Linear Regression.

The goal is to give a basic introduction to regression problems, so many things will be simplified.

The data is from the Ames Housing dataset, obtained through [Kaggle](https://www.kaggle.com/c/house-prices-advanced-regression-techniques/overview).

Regarding notation, $a$ is a scalar, $\mathbf{a}$ is a vector (column vector), and $\mathbf{A}$ is a matrix.

## Imports

```python
import matplotlib.pyplot as plt  # Plotting
import numpy as np  # Array computation
import pandas as pd  # Handling table data
import seaborn as sns  # Plotting
from IPython.display import display, clear_output  # Plotting
%matplotlib inline
sns.set_theme()
sns.set_style("ticks")
```

## Load and Process data
For simplicity, we will only consider the LotArea feature and the first 50 examples of the dataset for training.

```python
# Load and show full training data
train_data_frame = pd.read_csv("Data\\train.csv")
train_data_frame
```

      Id
      MSSubClass
      MSZoning
      LotFrontage
      LotArea
      Street
      Alley
      LotShape
      LandContour
      Utilities
      ...
      PoolArea
      PoolQC
      Fence
      MiscFeature
      MiscVal
      MoSold
      YrSold
      SaleType
      SaleCondition
      SalePrice

      0
      1
      60
      RL
      65.0
      8450
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      2
      2008
      WD
      Normal
      208500

      1
      2
      20
      RL
      80.0
      9600
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      5
      2007
      WD
      Normal
      181500

      2
      3
      60
      RL
      68.0
      11250
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      9
      2008
      WD
      Normal
      223500

      3
      4
      70
      RL
      60.0
      9550
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      2
      2006
      WD
      Abnorml
      140000

      4
      5
      60
      RL
      84.0
      14260
      Pave
      NaN
      IR1
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      12
      2008
      WD
      Normal
      250000

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      1455
      1456
      60
      RL
      62.0
      7917
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      8
      2007
      WD
      Normal
      175000

      1456
      1457
      20
      RL
      85.0
      13175
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      MnPrv
      NaN
      0
      2
      2010
      WD
      Normal
      210000

      1457
      1458
      70
      RL
      66.0
      9042
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      GdPrv
      Shed
      2500
      5
      2010
      WD
      Normal
      266500

      1458
      1459
      20
      RL
      68.0
      9717
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      4
      2010
      WD
      Normal
      142125

      1459
      1460
      20
      RL
      75.0
      9937
      Pave
      NaN
      Reg
      Lvl
      AllPub
      ...
      0
      NaN
      NaN
      NaN
      0
      6
      2008
      WD
      Normal
      147500

1460 rows × 81 columns

```python
train_data_frame = train_data_frame[["LotArea", "SalePrice"]]  # Select/Index only the LotArea and SalePrice columns
train_data_frame
```

      LotArea
      SalePrice

      0
      8450
      208500

      1
      9600
      181500

      2
      11250
      223500

      3
      9550
      140000

      4
      14260
      250000

      ...
      ...
      ...

      1455
      7917
      175000

      1456
      13175
      210000

      1457
      9042
      266500

      1458
      9717
      142125

      1459
      9937
      147500

1460 rows × 2 columns

```python
# Take the first 50 examples
train_data_frame = train_data_frame[:50]

# Scale the Sale Price and Lot Area
train_data_frame[["SalePrice"]] /= 1000
train_data_frame[["LotArea"]] /= 1000

# Create scatter plot
sns.relplot(x="LotArea", y="SalePrice", data=train_data_frame)
plt.title("Sale Prices")
plt.ylabel("Sale Price (thousands $)")
plt.xlabel("Lot Area (thousands sq. feet)")
```

Let's reshape our data so it's easier to work with.

We want train_xs to be a matrix with shape ($m$, $n + 1$) with the first column being all ones (in our case $m = 50$ and $n = 1$):
\begin{align}
\mathbf{X} = \begin{bmatrix} x^1_{0} && \ldots && x^1_{n} \\ \vdots && \ddots && \vdots \\ x^m_{0} && \ldots && x^m_{n} \end{bmatrix}
\end{align}

We want train_ys to be a vector with shape ($m$, 1):
\begin{align}
\mathbf{y} = \begin{bmatrix} y^1 \\ \vdots \\ y^m \end{bmatrix}
\end{align}

```python
train_xs = train_data_frame[["LotArea"]].to_numpy()
train_ys = train_data_frame[["SalePrice"]].to_numpy()
train_ys
```

    array([[208.5  ],
           [181.5  ],
           [223.5  ],
           [140.   ],
           [250.   ],
           [143.   ],
           [307.   ],
           [200.   ],
           [129.9  ],
           [118.   ],
           [129.5  ],
           [345.   ],
           [144.   ],
           [279.5  ],
           [157.   ],
           [132.   ],
           [149.   ],
           [ 90.   ],
           [159.   ],
           [139.   ],
           [325.3  ],
           [139.4  ],
           [230.   ],
           [129.9  ],
           [154.   ],
           [256.3  ],
           [134.8  ],
           [306.   ],
           [207.5  ],
           [ 68.5  ],
           [ 40.   ],
           [149.35 ],
           [179.9  ],
           [165.5  ],
           [277.5  ],
           [309.   ],
           [145.   ],
           [153.   ],
           [109.   ],
           [ 82.   ],
           [160.   ],
           [170.   ],
           [144.   ],
           [130.25 ],
           [141.   ],
           [319.9  ],
           [239.686],
           [249.7  ],
           [113.   ],
           [127.   ]])

```python
# Add extra feature to xs
train_xs = train_xs.reshape(50, 1)
ones = np.ones(shape=(50, 1))
train_xs = np.c_[ones, train_xs]  # Shape is now (50, 2)
train_xs
```

    array([[ 1.   ,  8.45 ],
           [ 1.   ,  9.6  ],
           [ 1.   , 11.25 ],
           [ 1.   ,  9.55 ],
           [ 1.   , 14.26 ],
           [ 1.   , 14.115],
           [ 1.   , 10.084],
           [ 1.   , 10.382],
           [ 1.   ,  6.12 ],
           [ 1.   ,  7.42 ],
           [ 1.   , 11.2  ],
           [ 1.   , 11.924],
           [ 1.   , 12.968],
           [ 1.   , 10.652],
           [ 1.   , 10.92 ],
           [ 1.   ,  6.12 ],
           [ 1.   , 11.241],
           [ 1.   , 10.791],
           [ 1.   , 13.695],
           [ 1.   ,  7.56 ],
           [ 1.   , 14.215],
           [ 1.   ,  7.449],
           [ 1.   ,  9.742],
           [ 1.   ,  4.224],
           [ 1.   ,  8.246],
           [ 1.   , 14.23 ],
           [ 1.   ,  7.2  ],
           [ 1.   , 11.478],
           [ 1.   , 16.321],
           [ 1.   ,  6.324],
           [ 1.   ,  8.5  ],
           [ 1.   ,  8.544],
           [ 1.   , 11.049],
           [ 1.   , 10.552],
           [ 1.   ,  7.313],
           [ 1.   , 13.418],
           [ 1.   , 10.859],
           [ 1.   ,  8.532],
           [ 1.   ,  7.922],
           [ 1.   ,  6.04 ],
           [ 1.   ,  8.658],
           [ 1.   , 16.905],
           [ 1.   ,  9.18 ],
           [ 1.   ,  9.2  ],
           [ 1.   ,  7.945],
           [ 1.   ,  7.658],
           [ 1.   , 12.822],
           [ 1.   , 11.096],
           [ 1.   ,  4.456],
           [ 1.   ,  7.742]])

## Hypothesis Definition
We now want to implement Linear Regression to predict *Sale Price* from *Lot Area*.

The hypothesis function $h$ is defined as follows for a single example $\mathbf{x}^k$ containing $n$ features:

\begin{align}
h_\theta(\mathbf{x}^k) & = \theta_0 + \theta_1x^k_1 + \theta_2x^k_2 + \ldots + \theta_nx^k_n \\
& = \sum_{i=0}^n \theta_ix^k_i \\
& = \begin{bmatrix} \theta_0 && \ldots && \theta_n \end{bmatrix} \begin{bmatrix} x^k_0 \\ \vdots \\ x^k_n \end{bmatrix} \\
& = \boldsymbol{\theta}^T\mathbf{x}^k
\end{align}

We want to implement a batched method which computes the hypothesis for all examples in one go. To do so, we can implement the following version:

\begin{align}
h_\theta(\mathbf{X}) & =  \mathbf{X} \boldsymbol{\theta} = \begin{bmatrix} x^1_{0} && \ldots && x^1_{n} \\ \vdots && \ddots && \vdots \\ x^m_{0} && \ldots && x^m_{n} \end{bmatrix} \begin{bmatrix} \theta_0 \\ \vdots \\ \theta_n \end{bmatrix}  = \begin{bmatrix} h_\theta(\mathbf{x}^1) \\ \vdots \\ h_\theta(\mathbf{x}^m) \end{bmatrix}
\end{align}

```python
# In our case we only need 2 thetas
parameters = np.array([[0.001], [0.001]])  # Shape is (2, 1)

def linear_hypothesis(thetas, input_data):
    """
    thetas: parameters for the hypothesis, expected shape is (2, 1)
    input_data: full matrix of input data, expected shape is (n_samples, 2)

    return: hypothesis result for each input sample, shape is (n_samples, 1)
    """
    return np.dot(input_data, thetas)

```

## Define Learning Algorithm
First step, define the cost function for the entire dataset:
\begin{align}
J_\theta(\mathbf{X}, \mathbf{y}) = \frac{1}{2}\sum_{k=1}^m (h_\theta(\mathbf{x}^k) - \mathbf{y}^k)^2
\end{align}

This standard definition sums over the squared error for each example in the set to produce a single scalar for the cost.

We can instead implement a vectorized form:
\begin{align}
J_\theta(\mathbf{X}, \mathbf{y}) = \frac{1}{2} (h_\theta(\mathbf{X}) - \mathbf{y})^T (h_\theta(\mathbf{X}) - \mathbf{y})
\end{align}

```python
def cost_function(thetas, input_data, target_data):
    """
    thetas: parameters for the hypothesis, expected shape is (2, 1)
    input_data: full matrix of input data, expected shape is (n_samples, 2)
    target_data: full vector of target data, expected shape is (n_samples, 1)

    return: total aggregate cost
    """
    hypothesis_result = linear_hypothesis(thetas, input_data)

    return (1/2) * np.dot((hypothesis_result - target_data).T, (hypothesis_result - target_data))[0][0]
```

Second step, define gradient descent per epoch, where each parameter $\theta_i$ is updated following:
\begin{align}
\theta_i & ≔ \theta_i - \alpha \frac{\partial}{\partial \theta_i} J_\theta(\mathbf{X}, \mathbf{y}) \\
& ≔ \theta_i - \alpha \sum_{k=1}^m (h_\theta(\mathbf{x}^k) - y^k)x^k_i
\end{align}
As before, we can instead implement a vectorized operation:

\begin{align}
\boldsymbol{\theta} & ≔ \boldsymbol{\theta} - \alpha \mathbf{X}^T(h_\theta(\mathbf{X}) - \mathbf{y})
\end{align}

```python
def gradient_descent(thetas, input_data, target_data, alpha=0.1):
    """
    thetas: parameters for the hypothesis, expected shape is (2, 1)
    input_data: full matrix of input data, expected shape is (n_samples, 2)
    target_data: full vector of target data, expected shape is (n_samples, 1)
    alpha: learning rate, defaults to 0.1

    return: New parameters, shape is (2, 1)
    """
    return thetas - alpha * np.dot(input_data.T, linear_hypothesis(thetas, input_data) - target_data)

```

## Train
Now we can train the model for multiple epochs and see how the result!

```python
# Let's redefine the parameters so we can try different starting points
parameters = np.array([[0.01], [0.01]])

# Create array of xs to use for plotting
x_space = np.linspace(0, 20, 500).reshape(500, 1)
ones = np.ones(shape=(500, 1))
x_added = np.c_[ones, x_space]

def create_fig():
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)

    return fig, ax

def plot(ax, fig, parameters):
    current_line = linear_hypothesis(parameters, x_added)

    ax.cla()
    ax.set_ylim([0, 400])
    ax.set_xlim([0, 20])
    ax.set_ylabel("Sale Price (thousands $)")
    ax.set_xlabel("Lot Area (thousands sq. feet)")
    ax.plot(x_space.flatten(), current_line.flatten(), "r--", label="Linear Regression estimate")
    ax.scatter(train_data_frame[["LotArea"]], train_data_frame[["SalePrice"]], label="Real data")
    ax.legend(loc="upper left")
    display(fig)

    clear_output(wait = True)
    plt.pause(0.5)

def train(parameters, n_epochs=15):
    # Create empty figure for plotting
    fig, ax = create_fig()

    # Iterate over n_epochs
    for epoch in range(n_epochs):
        parameters = gradient_descent(parameters, train_xs, train_ys, alpha=0.0001)
        plot(ax, fig, parameters)

    return parameters

trained_parameters = train(parameters)
```

```python
print(f"Final average training cost: {cost_function(trained_parameters, train_xs, train_ys):.2f}")
print(f"Final trained parameters: {trained_parameters}")
```

    Final average training cost: 111906.72
    Final trained parameters: [[ 2.01086818]
     [17.39416509]]

 # Library Version
Insted of implementing everything by hand, we can use a library.

The next cells show how to use the sklearn library for linear regression. Note that the Linear Regression implemented by the library uses the Ordinary Least Squares method instead of Gradient Descent.

This method returns the optimal parameters by setting the gradient of the cost function $J$ to 0:

\begin{align}
J_\theta(\mathbf{X}, \mathbf{y}) &= \frac{1}{2} (h_\theta(\mathbf{X}) - \mathbf{y})^T (h_\theta(\mathbf{X}) - \mathbf{y}) \\
\nabla_\theta J_\theta(\mathbf{X}, \mathbf{y}) &= \mathbf{X}^T\mathbf{X}\boldsymbol\theta - \mathbf{X}^T\mathbf{y} = 0 \\
\mathbf{X}^T\mathbf{X}\boldsymbol\theta &= \mathbf{X}^T\mathbf{y} \\
\boldsymbol\theta &= (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}
\end{align}

```python
from sklearn.linear_model import LinearRegression

regressor = LinearRegression()
regressor = regressor.fit(train_xs[:, 1:], train_ys)  # Remove the extra features from x
sklearn_params = np.array([[regressor.intercept_[0]], [regressor.coef_[0, 0]]])  # Write the learned parameters as before
fig, ax = create_fig()
plot(ax, fig, sklearn_params)
```

```python
print(f"Final average training cost: {cost_function(sklearn_params, train_xs, train_ys):.2f}")
print(f"Final trained parameters: {sklearn_params}")
```

    Final average training cost: 103494.53
    Final trained parameters: [[68.29682524]
     [11.22112049]]


# Incremental control example
Åström & Wittenmark Problem 5.3

We have plant model
$$ H(z) = \frac{z+0.7}{z^2 - 1.8z + 0.81} $$
and controller
$$ F_b(z) = \frac{s_0z^2 + s_1z + s_2}{(z-1)(z + r_1)} $$
Want closed-loop characteristic polynomial $A_c(z) = z^2 - 1.5z + 0.7$ and observer poles in the range $0<\alpha<a$.
### Diophantine equation
$$  (z^2 - 1.8z + 0.81)(z-1)(z+r_1) + (z+0.7)(s_0z^2 + s_1z + s_2) = (z-\alpha)^2(z^2-1.5z+0.7) $$
\begin{align}
(z^2 - 1.8z + 0.81)(z^2 +(r_1-1)z - r_1) + s_0z^3 + s_1z^2 + s_2z + 0.7s_0z^2 + 0.7s_1z + 0.7s_2 &= \\ \qquad\qquad\qquad\qquad\qquad\qquad (z^2 -2\alpha z + \alpha^2)(z^2 - 1.5z + 0.7)\\
(z^4 - 1.8z^3 + 0.81z^2 + (r_1-1)z^3 - 1.8(r_1-1)z^2 + 0.81(r_1-1)z - r_1z^2 + 1.8r_1z - 0.81r_1 + \\
\qquad\qquad\qquad s_0z^3 + s_1z^2 + s_2z + 0.7s_0z^2 + 0.7s_1z + 0.7s_2 = \\
\qquad\qquad\qquad\qquad\qquad\qquad z^4 - 2\alpha z^3 + \alpha^2 z^2 -1.5z^3 + 3\alpha z^2 - 1.5\alpha^2 z + 0.7z^2 - 1.4\alpha z + 0.7\alpha^2
\end{align}

Resulting equations when setting coefficients equal
\begin{align}
\begin{cases} z^3: & r_1 + s_0 = 1.8 + 1 -1.5 -2\alpha\\
              z^2: & -2.8 r_1 + 0.7s_0 + s_1 = -0.81 -1.8 +0.7 + \alpha^2 + 3\alpha\\
              z^1: &  2.61r_1 + 0.7s_1 + s_2 = 0.81 -1.5\alpha^2 - 1.4\alpha\\
              z^0: & -0.81r_1 + 0.7s_2 = 0.7\alpha^2  \end{cases}
\end{align}

### Feedforward part of controller
$$T(z) = t_0A_o(z) = t_0(z-\alpha)^2$$
$$ G_c(z) = \frac{T(z)B(z)}{A_o(z)A_c(z)} = \frac{t_0 B(z)}{A_c(z)}, \quad \text{want}\, G_c(1)=1$$
$$t_0 = \frac{A_c(1)}{B(1)} = \frac{1 - 1.5+0.7}{1+0.7} = \frac{2}{17}$$

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sy
import control.matlab as cm
import ipywidgets as widgets
%matplotlib widget
```

## Symbolic solution

```python
sy.init_printing()
alphaa, hh, r1, s0, s1,s2 = sy.symbols('alpha, h, r1, s0, s1,s2',
                                        real=True)
zz = sy.symbols('z', real=False)
```

```python
A = zz**2 - 1.8*zz + 0.81
B = zz+0.7
R = (zz-1)*(zz+r1)
S = s0*zz**2 + s1*zz + s2
LHS = sy.Poly(A*R + B*S, zz)
LHS
```

```python
RHS = sy.Poly((zz-alphaa)**2*(zz**2 - 1.5*zz + 0.7), zz)
Dioph = LHS-RHS
coeffs = Dioph.all_coeffs()
coeffs
```

```python
sol=sy.solve(coeffs, [r1,s0, s1,s2]);
```

## Effect of observer pole

```python
h = 1
# The plant
H = cm.tf([1, 0.7], [1, -1.8, 0.81], h)

# set up plot
fig, ax = plt.subplots(figsize=(8, 4))
#ax.set_ylim([-.1, 4])
ax.grid(True)

def bode_S_T_and_Gc(alpha):
    """
    Returns the bode plot data for S and T given observer pole
    """
    r1n = float(sol[r1].subs({alphaa: alpha}))
    s0n = float(sol[s0].subs({alphaa: alpha}))
    s1n = float(sol[s1].subs({alphaa: alpha}))
    s2n = float(sol[s2].subs({alphaa: alpha}))

    B = [1, 0.7]
    Ac = [1, -1.5, 0.7]
    Rp = np.convolve([1, r1n],[1,-1])
    Fb = cm.tf([s0n, s1n, s2n], Rp , h)
    Ff = np.sum(Ac)/np.sum(B) * cm.tf(np.convolve([1, -alpha], [1, -alpha]), Rp, h)

    #Gc = cm.minreal(Ff * cm.feedback(H, Fb))
    Gc = Ff * cm.feedback(H, Fb)
    Ss = cm.feedback(1, H*Fb)
    Ts = cm.feedback(H*Fb, 1)

    Gcbode = cm.bode(Gc, omega_limits=[0.01, 3], Plot=False)
    Sbode = cm.bode(Ss, omega_limits=[0.01, 3], Plot=False)
    Tbode = cm.bode(Ts, omega_limits=[0.01,3], Plot=False)

    return (Sbode, Tbode, Gcbode)

@widgets.interact(alpha=(0, 0.9, 0.02))
def update(alpha = 0.4):
    """Remove old lines from plot and plot new one"""
    [l.remove() for l in ax.lines]
    [l.remove() for l in ax.lines]
    Sb, Tb, Gb = bode_S_T_and_Gc(alpha)
    sh = ax.loglog(Sb[2], Sb[0], color='C0', label='Ss')
    th = ax.loglog(Tb[2], Tb[0], color='C1', label='Ts')
    gh = ax.loglog(Gb[2], Gb[0], color='C2', label='Gc')
    ax.legend(loc='lower center')

```

```python
h = 1
# The plant
H = cm.tf([1, 0.7], [1, -1.8, 0.81], h)

# set up plot
fig, ax = plt.subplots(3,1,figsize=(8, 6), sharex=True )
ax[0].set_ylabel('Resp to pert')
ax[1].set_ylabel('Resp to noise')
ax[2].set_ylabel('Resp to setpoint')
def step_S_T_and_Gc(alpha):
    """
    Returns the bode plot data for S and T given observer pole
    """
    r1n = float(sol[r1].subs({alphaa: alpha}))
    s0n = float(sol[s0].subs({alphaa: alpha}))
    s1n = float(sol[s1].subs({alphaa: alpha}))
    s2n = float(sol[s2].subs({alphaa: alpha}))

    B = [1, 0.7]
    Ac = [1, -1.5, 0.7]
    Rp = np.convolve([1, r1n],[1,-1])
    Fb = cm.tf([s0n, s1n, s2n], Rp , h)
    Ff = np.sum(Ac)/np.sum(B) * cm.tf(np.convolve([1, -alpha], [1, -alpha]), Rp, h)

    #Gc = cm.minreal(Ff * cm.feedback(H, Fb))
    Gc = Ff * cm.feedback(H, Fb)
    Ss = cm.feedback(1, H*Fb)
    Ts = cm.feedback(H*Fb, 1)

    NN = 50
    uu = np.ones(NN)
    uu[24:] = 0
    tt = np.arange(NN)
    Gcstep = cm.lsim(Gc, uu, tt)
    Sstep = cm.lsim(Ss, uu, tt)
    Tstep = cm.lsim(Ts, uu, tt)

    return (Sstep, Tstep, Gcstep)

@widgets.interact(alpha=(0, 0.9, 0.02))
def update(alpha = 0.4):
    """Remove old lines from plot and plot new one"""
    [[l.remove() for l in ax_.lines] for ax_ in ax]
    [[l.remove() for l in ax_.lines] for ax_ in ax]
    Sb, Tb, Gb = step_S_T_and_Gc(alpha)
    sh = ax[0].stem(Sb[1], Sb[0], markerfmt='C0o', label='Ss')
    th = ax[1].stem(Tb[0], markerfmt='C1o', label='Ts')
    gh = ax[2].stem(Gb[0], markerfmt='C2o', label='Gc')

```

```python
?plt.subplots
```

```python

```

```python
h = 1
# The plant
H = cm.tf([1, 0.7], [1, -1.8, 0.81], h)

# set up plot
fig, ax = plt.subplots(figsize=(8, 4))
#ax.set_ylim([-.1, 4])
ax.grid(True)

def bode_S_T_and_Gc(alpha):
    """
    Returns the bode plot data for S and T given observer pole
    """
    r1n = float(sol[r1].subs({alphaa: alpha}))
    s0n = float(sol[s0].subs({alphaa: alpha}))
    s1n = float(sol[s1].subs({alphaa: alpha}))
    s2n = float(sol[s2].subs({alphaa: alpha}))

    B = [1, 0.7]
    Ac = [1, -1.5, 0.7]
    Rp = np.convolve([1, r1n],[1,-1])
    Fb = cm.tf([s0n, s1n, s2n], Rp , h)
    Ff = np.sum(Ac)/np.sum(B) * cm.tf(np.convolve([1, -alpha], [1, -alpha]), Rp, h)

    #Gc = cm.minreal(Ff * cm.feedback(H, Fb))
    Gc = Ff * cm.feedback(H, Fb)
    Ss = cm.feedback(1, H*Fb)
    Ts = cm.feedback(H*Fb, 1)

    Gcbode = cm.bode(Gc, omega_limits=[0.01, 3], Plot=False)
    Sbode = cm.bode(Ss, omega_limits=[0.01, 3], Plot=False)
    Tbode = cm.bode(Ts, omega_limits=[0.01,3], Plot=False)

    return (Sbode, Tbode, Gcbode)

@widgets.interact(alpha=(0, 0.9, 0.02))
def update(alpha = 0.4):
    """Remove old lines from plot and plot new one"""
    [l.remove() for l in ax.lines]
    [l.remove() for l in ax.lines]
    Sb, Tb, Gb = bode_S_T_and_Gc(alpha)
    sh = ax.semilogy(Sb[2], Sb[0], color='C0', label='Ss')
    th = ax.semilogy(Tb[2], Tb[0], color='C1', label='Ts')
    gh = ax.semilogy(Gb[2], Gb[0], color='C2', label='Gc')
    ax.legend(loc='lower center')

```

    FigureCanvasNbAgg()

    interactive(children=(FloatSlider(value=0.4, description='alpha', max=0.9, step=0.02), Output()), _dom_classes…

```python

```