# Analytic 1D Solution

Allow the following definitions/assumptions:

1. The sky is has a uniform distribution of point sources that follow a power-law source count spectrum. This distribution is uniform in (l,m) for simplicity.
2. Each antenna/tile has the same frequency-independent Gaussian beam.
3. Each antenna/tile has the same bandpass, whose multiplication with a frequency taper we will call $\phi$.
4. The frequency channels are fine enough* that it is a reasonable approximation to represent the visibility in each channel as the instantaneous value at the central frequency, rather than the average throughout the channel.
5. All baselines lie along a 1D slice.
6. Let $\nu_0$ be a reference frequency, and also the central frequency of the band (at which $\phi(\nu_0) = 1$).
7. Let $f = \nu/\nu_0$ be a normalised frequency.
8. Let $\omega = \nu_0 \eta$ be a unitless fourier-dual to frequency.
9. Let all _subscripted_ $u$ be defined as the value of $u$ for a _specific baseline_ at the reference frequency, so that the actual value of $u$ for baseline $i$ is $f u_i$.
10. Let the sampled frequencies be $f_k$, where $k \in (-n_f/2, n_f/2)$, and $f_0 = 1$.

*We'll probably have to think about what defines "fine enough" in this context later.

## One Baseline per Frequency Approximation

Slightly more controversially, let us consider the case in which the baselines are either completely redundant, or they are spaced widely enough that it is _almost always_ reasonable to approximate the value of the visibility at $u$ by the value of the floating visibility at the closest $u_i$. This imposes the limit that the average distance between baselines at this particular $u$ (and for the range of frequencies involved), is much larger than about $1/2\pi\sigma \approx 1$. If this is not the case, then the maths becomes significantly more difficult.

Then the expecation of the power spectrum is basically proportional to the variance of the visibility at $u, \omega$, which can be expressed as:

\begin{equation}
    {\rm Var}(V(u,\omega))  = \pi \sigma^2 \nu_0 \mu_2 \Delta_f^2 \sum_{j,k = -n_f/2}^{n_f/2} \phi_j\phi_k e^{-\pi^2\sigma^2 (f_j \hat{u}_j - f_k \hat{u}_k)^2} e^{-2\pi i \omega(f_j - f_k)},
\end{equation}

where $\hat{u}_j$ is defined as the value of $u_i$ for the baseline closest to $u$ at $f_j$.

Some simple cases follow:

### Sparse Baselines

If baselines are _very widely_ spaced, such that $\hat{u}_{-n_f/2} = \hat{u}_{n_f/2} \approx u$, then we achieve

\begin{equation}
    {\rm Var}(V(u,\omega))  = \pi \sigma^2 \nu_0 \mu_2 \Delta_f^2 \sum_{j,k = -n_f/2}^{n_f/2} \phi_j\phi_k e^{-\pi^2\sigma^2 u^2 (f_j - f_k )^2} e^{-2\pi i \omega(f_j - f_k)}.
\end{equation}

which can be seen to be exactly analogous to the archetypical wedge case (see my wedge paper, Table 3).

### Total Redundancy

If baselines are spaced in any way such that the closest baseline is literally at $u$ for any frequency channel, then $f_j u_j \equiv u$, and the exponential term drops out, leaving a simple solution: ${\rm Var}(V) \propto \left|\hat{\phi}\right|^2(\omega)$.

For this particular $u$, assuming that there exists a baseline at $u$ for $f_0$, then we require _at least_ the baselines $u_i = u/(1 + i \Delta_f)$ for $i \in (-n_f/2, n_f/2)$ for _this particular_ $u$.

To evaluate a set of scales $u_\mu$ ($\mu < N_u$)  would thus require the set $\{u_\mu/(1 + i \Delta_f)\}$ which is no larger than $N_u n_f$, but may be smaller if suitable values of $u_\mu$ can be chosen so that baselines can coincide.

Problematically, given a standard observation setup of ~1000 channels over 100MHz, so that $\Delta_f \approx 0.1/150 = 6\times10^{-4}$, the relevant $u_i$ for say $u=50$, are spaced by $\Delta_u \approx 0.033$. Note that this is not a _precision_ of placement, but the necessary _density_ of baselines.

Possibly more problematically, at this density, the assumption that only the closest baseline counts is broken, and the entire equation doesn't hold any more. It is unclear to me at this point what the ramifications for this on the wedge are. The finest frequency channels allowable in order to maintain the assumption for the largest scales of interest ($u \sim 150$), are of the order of $\Delta_f = 1/150 = 6 \times 10^{-3}$, which correspond to 1MHz. But this limits the window to smaller $\omega$

### Partial Redundancy

What would happen if every $n^{th}$ frequency was completely redundant (at least, every $n^{th}$ had a baseline at $u$, and there were no other baselines in-between)? Then for the previous $n/2$ frequencies, and following $n/2$, the $\hat{u}_j$ would be the same

### Numerical Evaluation

```python
import numpy as np
from powerbox.dft import fftfreq, fft
import matplotlib.pyplot as plt
from multiprocessing import cpu_count
import pickle
import ctypes

from plot_tools import suplabel, make_plot

cdll = ctypes.CDLL("./varvis.so")

%matplotlib notebook

%load_ext autoreload
%autoreload 2
```

    /home/steven/miniconda3/envs/pssim/lib/python3.6/site-packages/powerbox/dft.py:51: UserWarning: You do not have pyFFTW installed. Installing it should give some speed increase.
      warnings.warn("You do not have pyFFTW installed. Installing it should give some speed increase.")

```python
def get_f_omega(numin, numax, nf, nomega=50, log_omega=False):
    nu0 = (numin + numax) / 2
    f = np.linspace(numin / nu0, numax / nu0, nf)

    omega = fftfreq(len(f), d=f[1] - f[0], b=2 * np.pi)
    omega = omega[omega > 0]
    if log_omega:
        omega = np.logspace(np.log10(omega.min()), np.log10(omega.max()), nomega)

    return f, omega
```

```python
# COMMON PARAMETERS
numin = 125
numax = 175
nf = 250

f, omega = get_f_omega(numin, numax, nf)

bl_nums = [100, 200, 300, 500]
u_points = [15, 50, 150]

bl_min = 3
bl_max = 250

taper = np.blackman
sigma = 0.2

log_omega = False
```

```python
def var_vis_one_bl_per_freq(u, ui, f, omega, taper, sigma):
    """
    Evaluate the variance of visibility at u given baselines ui and frequencies f.

    Does so under the approximation that at any frequency, only the closest baseline
    to u actually contributes.
    """
    # Get closest baselines
    uj = np.zeros_like(f)
    for i,ff in enumerate(f):
        uj[i] = ff * ui[np.argmin(np.abs(u - ff*ui))]

    exp_part = np.exp(-np.pi**2*sigma**2*np.add.outer(uj,-uj)**2)

    bp = np.outer(taper(len(f)), taper(len(f)))
    bp *= exp_part
    omegaff = np.outer(omega, np.add.outer(f,-f)).reshape((len(omega), len(f), len(f)))

    kernel = np.exp(-2j*np.pi*omegaff)

    return np.abs(np.sum(kernel*bp, axis=(1,2)))

def var_vis_perfect_redundancy(f, omega, taper, sigma):
    """
    Determine the variance of visibilities in the limit of perfect redundancy, _and_
    that baselines are separated enough to allow the one_bl_per_freq approximation.
    """
    bp = np.outer(taper(len(f)), taper(len(f)))
    omegaff = np.outer(omega, np.add.outer(f,-f)).reshape((len(omega), len(f), len(f)))

    kernel = np.exp(-2j*np.pi*omegaff)

    return np.abs(np.sum(kernel*bp, axis=(1,2)))
```

## Denser Baselines

In the event that we can pack baselines closer than $\Delta \approx 1$ (and we'll need to do this to eliminate the wedge), then the approximation above will not work. **That is, we should pretty much just completely ignore the above.**

In this case we begin from the following:

\begin{align}
		{\rm Var}(\tilde{V}) &= \mu_2\nu_0^2 \int d^2\vec{l}\  e^{-l^2/\sigma^2} |I|^2, \\
		I &= \Delta_f \sum_{j=-nf/2}^{n_f/2}\ \frac{\phi_\nu}{W_\nu(\vec{u})}  \sum_{i=1}^{N_{\rm bl}} B(\vec{u} - f\vec{u}_i)e^{-2\pi i f(\omega + \vec{l}\cdot\vec{u}_i)} \nonumber \\
		&= \Delta_f \sum_{j=-nf/2}^{n_f/2}\frac{\phi_\nu \sum_{i=1}^{N_{\rm bl}} e^{-2\pi^2\sigma^2(\vec{u}-f\vec{u}_i)^2} e^{-2\pi i f(\omega + \vec{l}\cdot\vec{u}_i)}}{\sum_{i=1}^{N_{\rm bl}} e^{-2\pi^2\sigma^2(\vec{u}-f\vec{u}_i)^2}} \\
        W_\nu(u) &= \sum_{i=1}^{N_{\rm bl}} e^{-2\pi^2\sigma^2(\vec{u}-f\vec{u}_i)^2}
\end{align}

Doing the $l$ integral, we assume that all $u$ lie along the $x$-axis, which gives

\begin{align}
{\rm Var}(\tilde{V}) &= \sqrt{\pi \sigma^2} \mu_2\nu_0^2 \Delta^2_f
		\sum_{j,k=-n_f/2}^{n_f/2}  \frac{\phi_j\phi_k e^{-2\pi i \omega (f_j-f_k)}}{W_j(u)W_k(u)}\sum_{i,m=1}^{N_{\rm bl}} e^{-2\pi^2\sigma^2(\vec{u}-f_ju_i)^2} e^{-2\pi^2\sigma^2(u-f_ku_m)^2} \int d l_y\  e^{-l_y^2/\sigma^2} e^{-2\pi i l_y( f_j u_i - f_ku_m)}  \\
        &= \pi \sigma^2 \mu_2\nu_0^2 \Delta^2_f
		\sum_{j,k=-n_f/2}^{n_f/2}  \frac{\phi_j\phi_k e^{-2\pi i \omega (f_j-f_k)}}{W_j(u)W_k(u)}\sum_{i,m=1}^{N_{\rm bl}} e^{-2\pi^2\sigma^2(u-f_ju_i)^2} e^{-2\pi^2\sigma^2(u-f_ku_m)^2}  e^{-\pi^2\sigma^2 ( f_j u_i - f_ku_m)^2}.
\end{align}

Note that this reduces to the single-baseline-per-frequency case when we assume that it applies.

### Numerical Evaluation

```python
def var_vis_general(u, ui, f, omega, taper, sigma, extent=5, nthreads=None):
    """
    Evaluate the variance of a visibility at u given baselines ui and frequencies f.

    All baselines are used in the weighted average, as long as they are with `extent` by the kernel width.
    """
    if not nthreads:
        nthreads = cpu_count()

    cdll.var_vis.restype = ctypes.c_int
    cdll.var_vis.argtypes = [
        ctypes.c_int, ctypes.c_int, ctypes.c_int,
        np.ctypeslib.ndpointer(np.float64), np.ctypeslib.ndpointer(np.float64), np.ctypeslib.ndpointer(np.float64),
        np.ctypeslib.ndpointer(np.float64), ctypes.c_double, ctypes.c_double, ctypes.c_double,
        np.ctypeslib.ndpointer(np.float64), ctypes.c_int
    ]

    res = np.zeros(len(omega))
    taper = taper(len(f))

    cdll.var_vis(
        len(f), len(omega), len(ui),
        np.ascontiguousarray(f), np.ascontiguousarray(omega), np.ascontiguousarray(ui),
        np.ascontiguousarray(taper), sigma, u, extent, res, nthreads)

    return res
```

```python
# Actually perform calculations
expected_power = {}

# Get perfectly redundant solution
expected_power['omega'] = omega
expected_power['perfect'] = var_vis_perfect_redundancy(f, omega, taper, sigma)

for kind in ['lin', 'log']:
    expected_power[kind] = {}

    for i,n in enumerate(bl_nums):
        if kind=='log':
            ui = np.logspace(np.log10(bl_min), np.log10(bl_max), n)
        else:
            ui = np.linspace(bl_min, bl_max, n)

        for uu in u_points:
            print(f"Doing n={n}, u={uu}, {kind}")
            expected_power[kind][f'n={n} u={uu}'] = var_vis_general(uu, ui, f, omega, taper, sigma)

            # Dump results after every iter so we don't lose them.
            with open("expected_power.pkl", 'wb') as f:
                pickle.dump(expected_power, f)
```

    Doing n=100, u=15, lin
    Doing n=100, u=50, lin
    Doing n=100, u=150, lin
    Doing n=200, u=15, lin
    Doing n=200, u=50, lin
    Doing n=200, u=150, lin
    Doing n=300, u=15, lin
    Doing n=300, u=50, lin
    Doing n=300, u=150, lin
    Doing n=500, u=15, lin
    Doing n=500, u=50, lin
    Doing n=500, u=150, lin
    Doing n=100, u=15, log
    Doing n=100, u=50, log
    Doing n=100, u=150, log
    Doing n=200, u=15, log
    Doing n=200, u=50, log
    Doing n=200, u=150, log
    Doing n=300, u=15, log
    Doing n=300, u=50, log
    Doing n=300, u=150, log
    Doing n=500, u=15, log
    Doing n=500, u=50, log
    Doing n=500, u=150, log

```python
make_plot(expected_power)
```

### Plots of weights

```python
def get_weight_at_u(u, sigma, ui):
    weight_i = np.exp(-2*np.pi**2*sigma**2*np.add.outer(u,-ui)**2)
    return np.sum(weight_i, axis=1)

def get_weights_f(u, ui, numin, numax, nf, sigma):
    # u, ui and f are arrays

    f = get_f_omega(numin, numax, nf)[0]

    wuf = np.zeros((len(u), len(f)))
    for i, ff in enumerate(f):
        wuf[:,i] = get_weight_at_u(u, sigma, ff*ui)

    return f, wuf

def get_weights_om(wuf, f, taper):
    wuom, omega = fft(wuf * taper(len(f)), L=f[-1] - f[0], axes=(1,))

    omega = omega[0]
    wuom = wuom[:, omega>0]
    omega = omega[omega>0]

    return omega, np.abs(wuom)**2
```

```python
# Create all weights
weights_f = {}
weights_om = {}

weights_om['perfect'] = var_vis_one_bl_per_freq(15, None, numin, numax, nf, taper, sigma)[1]

for kind in ['lin', 'log']:
    weights_f[kind] = {}
    weights_om[kind] = {}

    for i,n in enumerate(bl_nums):
        if kind=='log':
            ui = np.logspace(np.log10(bl_min), np.log10(bl_max), n)
        else:
            ui = np.linspace(bl_min, bl_max, n)

        f, wuf = get_weights_f(np.array(u_points), ui, numin, numax, nf, sigma)
        omega, wuom = get_weights_om(wuf, f, taper)

        if "omega" not in weights_f:
            weights_f['omega'] = omega
            weights_f['f'] = f
            weights_om['omega'] = omega
            weights_om['f'] = f

        for iu, uu in enumerate(u_points):
            weights_f[kind][f'n={n} u={uu}'] = wuf[iu]
            weights_om[kind][f'n={n} u={uu}'] = wuom[iu]

```

```python
make_plot(weights_om, normalise_at=0, supy=r"$W_u$")
```

```python
make_plot(weights_f, xscale_log=False, xquant='f', plot_perfect=False, supx=r"$f = \nu/\nu_0$", supy=r"$W_u(f)$")
```


## Variational Quantum Eigensolver (VQE) Example

### Table of Contents
1. [Introduction](#Introduction)
2. [Qubits, gates, and all that](#Qubits,-gates,-and-all-that)
3. [Make that Hamiltonian!](#Make-that-Hamiltonian!)
4. [A first attempt at a quantum circuit](#A-first-attempt-at-a-quantum-circuit)
5. [A "real" quantum circuit](#A-"real"-quantum-circuit)
6. [A "real" measurement of the energy](#A-"real"-measurement-of-the-energy)
7. [Appendix: All together now](#Appendix:-All-together-now)

### Introduction

The [variational quantum eigensolver (VQE)](https://en.wikipedia.org/wiki/Quantum_algorithm#Variational_quantum_eigensolver) is a hybrid classical-quantum algorithm that variationally determines the ground state energy of a Hamiltonian.

It's quantum in the sense that the expectation value of the energy is computed via a quantum algorithm, but it is classical in the sense that the energy is minimized with a classical optimization algorithm.

From a molecular electronic structure perspective, it is equivalent to computing the Full Configuration Interaction (FCI) for a given basis.

Quantum computing can be a little unintuitive, so it helps to see a working example. Although we won't actually do any "real" quantum computing, the methods can be understood from a linear algebra perspective. So that's what we are going to try and do for the VQE.

The problem we want to tackle is to implement the VQE to compute the energy of molecular hydrogen (H$_2$) in a minimal basis. We will base this implementation off the really neat paper

> O’Malley, Peter JJ, et al. "[Scalable quantum simulation of molecular energies.](https://link.aps.org/doi/10.1103/PhysRevX.6.031007)" Physical Review X 6.3 (2016): 031007.

In the O'Malley paper, they implement the VQE on a real quantum computer to compute the potential energy surface of H$_2$. The schematic we are going to follow can be seen below, and we are going to implement the "software".

Here's the big, overarching plan:

1. Put the Hamiltonian in the computational (qubit) basis.
2. Obtain a variational ansatz to parameterize the wave function.
3. Represent this ansatz as a quantum circuit.
4. Given this circuit, measure the expectation value of Hamiltonian (energy).
5. Vary the circuit parameters until the energy is minimized.

We'll look at these in turn, but first, let's look at single qubit states and some common matrices used to operate on them. This will set the groundwork for building up our VQE procedure.

### Qubits, gates, and all that

First, let's define some of the common quantum operators, gates, and states we want to work with. For starters, here are the identity and Pauli spin matrices:

$$
\mathbf{I} = \begin{pmatrix}
1 & 0 \\
0 & 1
\end{pmatrix}\quad
{X} = \begin{pmatrix}
0 & 1 \\
1 & 0
\end{pmatrix}\quad
{Y} = \begin{pmatrix}
0 & -i \\
i & 0
\end{pmatrix}\quad
{Z} = \begin{pmatrix}
1 & 0 \\
0 & -1
\end{pmatrix}
$$

Two other important gates are the Hadamard matrix and phase matrix:

$$
{H} = \frac{1}{\sqrt{2}}\begin{pmatrix}
1 & 1 \\
1 & -1
\end{pmatrix}\quad
{S} = \begin{pmatrix}
1 & 0 \\
0 & i
\end{pmatrix}
$$

Since for a single qubit the two basis states are
$$
|\psi\rangle = \begin{pmatrix}
|0\rangle\\
|1\rangle
\end{pmatrix}, \quad \mathrm{meaning} \qquad
|0\rangle = \begin{pmatrix}
1\\
0
\end{pmatrix}\quad
|1\rangle = \begin{pmatrix}
0\\
1
\end{pmatrix}
$$

we can define projection matrices, which are useful for defining, among other things, controlled gates (like CNOT, which we will see in a bit).

$$
|0\rangle\langle 0| = \begin{pmatrix}
1 & 0 \\
0 & 0
\end{pmatrix}\quad
|1\rangle\langle 1| = \begin{pmatrix}
0 & 0\\
0 & 1
\end{pmatrix}
$$

Also useful for single qubits, are the rotation matrices:

$$
\begin{align}
{R_x(\theta)} &= \begin{pmatrix}
\mathrm{cos}(\theta/2) & -i\cdot\mathrm{sin}(\theta/2) \\
-i\cdot\mathrm{sin}(\theta/2) & \mathrm{cos}(\theta/2)
\end{pmatrix}\\
{R_y(\theta)} &= \begin{pmatrix}
\mathrm{cos}(\theta/2) & -\mathrm{sin}(\theta/2) \\
\mathrm{sin}(\theta/2) & \mathrm{cos}(\theta/2)
\end{pmatrix}\\
{R_z(\theta)} &= \begin{pmatrix}
\mathrm{exp}(-i\theta/2) & 0 \\
0 & \mathrm{exp}(i\theta/2)
\end{pmatrix}
\end{align}
$$

So putting all these together for future use, we have:

```python
import numpy as np
np.set_printoptions(precision=4,suppress=True)

# Pauli matrices
I  = np.array([[ 1, 0],
               [ 0, 1]])
Sx = np.array([[ 0, 1],
               [ 1, 0]])
Sy = np.array([[ 0,-1j],
               [1j, 0]])
Sz = np.array([[ 1, 0],
               [ 0,-1]])

# Hadamard matrix
H = (1/np.sqrt(2))*np.array([[ 1, 1],
                             [ 1,-1]])

# Phase matrix
S = np.array([[ 1, 0],
              [ 0,1j]])

# single qubit basis states |0> and |1>
q0 = np.array([[1],
               [0]])
q1 = np.array([[0],
               [1]])

# Projection matrices |0><0| and |1><1|
P0  = np.dot(q0,q0.conj().T)
P1  = np.dot(q1,q1.conj().T)

# Rotation matrices as a function of theta, e.g. Rx(theta), etc.
Rx = lambda theta : np.array([[    np.cos(theta/2),-1j*np.sin(theta/2)],
                              [-1j*np.sin(theta/2),    np.cos(theta/2)]])
Ry = lambda theta : np.array([[    np.cos(theta/2),   -np.sin(theta/2)],
                              [    np.sin(theta/2),    np.cos(theta/2)]])
Rz = lambda theta : np.array([[np.exp(-1j*theta/2),                0.0],
                              [                0.0, np.exp(1j*theta/2)]])

```

Next, using these single qubit operations, we can build up some common two-qubit states and operations. By and large, moving from single to multiple qubits just involves tensor products of single qubit states and gates. Note that we number our qubits from bottom to top in this example, **so qubit-1 is on top and qubit-0 is on bottom**; e.g. in the figure:

So for our two qubits the basis looks like

$$
|\psi\rangle = |\hbox{qubit-1}\rangle \otimes |\hbox{qubit-0}\rangle =
\begin{pmatrix}
|0\rangle\otimes|0\rangle\\
|0\rangle\otimes|1\rangle\\
|1\rangle\otimes|0\rangle\\
|1\rangle\otimes|1\rangle\\
\end{pmatrix} =
\begin{pmatrix}
|00\rangle\\
|01\rangle\\
|10\rangle\\
|11\rangle\\
\end{pmatrix}
$$

so that, e.g.

$$
\begin{align}
|00\rangle &= \begin{pmatrix}
1\\
0\\
\end{pmatrix} \otimes \begin{pmatrix}
1\\
0\\
\end{pmatrix} = \begin{pmatrix}
1\\
0\\
0\\
0\\
\end{pmatrix},\\
|01\rangle &=
\begin{pmatrix}
1\\
0\\
\end{pmatrix} \otimes \begin{pmatrix}
0\\
1\\
\end{pmatrix} = \begin{pmatrix}
0\\
1\\
0\\
0\\
\end{pmatrix}, \mathrm{~etc.}
\end{align}
$$

For two qubits, useful are the CNOT$_{01}$ and CNOT$_{10}$, where the first digit is the "control" qubit and the second is the "target" qubit. (Again, recall that in the O'Malley paper, we define qubit-1 as the "top" qubit, and qubit-0 as the "bottom" qubit, so if you change the qubit numbering the definitions will be swapped).

They can be defined as the sum of two tensor products:

$$
\begin{align}
\mathrm{CNOT}_{10} &= (|0\rangle\langle 0| \otimes \mathbf{I}) + (|1\rangle\langle 1| \otimes {X}) = \begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 0 & 1\\
0 & 0 & 1 & 0
\end{pmatrix}\\
\mathrm{CNOT}_{01} &= (\mathbf{I} \otimes |0\rangle\langle 0|) + (X \otimes |1\rangle\langle 1|) = \begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 0 & 0 & 1\\
0 & 0 & 1 & 0\\
0 & 1 & 0 & 0
\end{pmatrix}\\
\end{align}
$$

The SWAP gate does what you'd expect, swapping qubit 1 and qubit 0:

$$
\mathrm{SWAP} = \begin{pmatrix}
1 & 0 & 0 & 0\\
0 & 0 & 1 & 0\\
0 & 1 & 0 & 0\\
0 & 0 & 0 & 1
\end{pmatrix}\\
$$

Most other operations are just simple tensor products of the single qubit operations. Like if you want to apply a Pauli X gate to qubit 1 and a Pauli Y gate to qubit 0, it's just $X_1 \otimes Y_0$:

$$
{X_1 \otimes Y_0} =
\begin{pmatrix}
0 & 1 \\
1 & 0 \\
\end{pmatrix}\otimes
\begin{pmatrix}
0 & -i \\
i & 0 \\
\end{pmatrix} =
\begin{pmatrix}
0 & 0 & 0 & -i\\
0 & 0 & i & 0\\
0 & -i & 0 & 0\\
i & 0 & 0 & 0\\
\end{pmatrix}\\
$$

Okay, let's get some of these coded up for later, and if we need any products, we know we can just `np.kron` them later as needed

```python
from scipy.linalg import block_diag

# CNOTij, where i is control qubit and j is target qubit
CNOT10 = np.kron(P0,I) + np.kron(P1,Sx) # control -> q1, target -> q0
CNOT01 = np.kron(I,P0) + np.kron(Sx,P1) # control -> q0, target -> q1

SWAP   = block_diag(1,Sx,1)
```

### Make that Hamiltonian!

Now we have our building blocks, we need to think about how to represent the Hamiltonian, which is usually in the fermion basis, in the basis of qubits. There are a few ways to do this, but the most common are the [Jordan-Wigner (JW)](https://en.wikipedia.org/wiki/Jordan%E2%80%93Wigner_transformation) transformation and the [Bravyi-Kitaev (BK)](https://arxiv.org/abs/1208.5986) transformation.

According to the O'Malley paper, using the BK-transformed Hamiltonian and exploiting some of the symmetry in the H$_2$ molecule, the Hamiltonian can be represented by only two qubits. (Nice.) The Hamiltonian has five components:

$$
\hat{H}_{\mathrm{BK}} = g_0 \mathbf{I} + g_1 Z_0 + g_2 Z_1 + g_3 Z_0Z_1 + g_4 Y_0Y_1 + g_5 X_0 X_1
$$

It general, the coefficients $g_i$ can be obtained from a cheap [Hartree-Fock](https://en.wikipedia.org/wiki/Hartree%E2%80%93Fock_method) calculation.

So let's build a Hamiltonian for the H$_2$ molecule with a bond length of 0.75 A. We will also also add the nuclear repulsion energy `nuclear_repulsion`. The parameters we use can be found in Table 1 of the Appendix in the O'Malley [paper](https://journals.aps.org/prx/pdf/10.1103/PhysRevX.6.031007), though for the nuclear repulsion energy you have to calculate yourself (I used Gaussian).

```python
# See DOI: 10.1103/PhysRevX.6.031007
# Here, we use parameters given for H2 at R=0.75A
g0 = -0.4804
g1 = +0.3435
g2 = -0.4347
g3 = +0.5716
g4 = +0.0910
g5 = +0.0910

nuclear_repulsion = 0.7055696146
```

With all this, we can build the Hamiltonian, `Hmol`, explicitly in matrix form by taking tensor products, also known as Kronecker products (using `np.kron`), of the single-qubit matrix operators we built previously.

```python
Hmol = (g0 * np.kron( I, I) + # g0 * I
        g1 * np.kron( I,Sz) + # g1 * Z0
        g2 * np.kron(Sz, I) + # g2 * Z1
        g3 * np.kron(Sz,Sz) + # g3 * Z0Z1
        g4 * np.kron(Sy,Sy) + # g4 * Y0Y1
        g5 * np.kron(Sx,Sx))  # g5 * X0X1
```

And let's take a look at the Hamiltonian matrix:

```python
print(Hmol)
```

    [[ 0.0000+0.j  0.0000+0.j  0.0000+0.j  0.0000+0.j]
     [ 0.0000+0.j -1.8302+0.j  0.1820+0.j  0.0000+0.j]
     [ 0.0000+0.j  0.1820+0.j -0.2738+0.j  0.0000+0.j]
     [ 0.0000+0.j  0.0000+0.j  0.0000+0.j  0.1824+0.j]]

Since we have the Hamiltonian in the computational basis, let's just diagonalize it to get the energy (lowest eigenvalue). By adding the nuclear repulsion energy to the result we should get the same result as a Full Configuration Interaction (FCI) calculation.

```python
electronic_energy = np.linalg.eigvalsh(Hmol)[0] # take the lowest value
print("Classical diagonalization: {:+2.8} Eh".format(electronic_energy + nuclear_repulsion))
print("Exact (from G16):          {:+2.8} Eh".format(-1.1457416808))
```

    Classical diagonalization: -1.1456295 Eh
    Exact (from G16):          -1.1457417 Eh

Considering that the Hamiltonian elements had a precision of 1E-04, this is very good agreement. However, this approach utilizes a classical algorithm to obtain the eigenvalues. We want to see if we can obtain the eigenvalues using a quantum circuit.

### A first attempt at a quantum circuit

The usual input for quantum algorithms is to start in the $|00\cdots\rangle$ state. This is represented by a zero vector, with the first element set to 1. Because our Hamiltonian for H$_2$ only requires two qubits, we will start with the state $|01\rangle$. To obtain this from $|00\rangle$, we just need to act on the zeroth qubit with the Pauli X operator. This is the first step in the quantum circuit in the figure I showed above from the O'Malley paper. (They apply X$_{\pi}$ -- same thing.)

```python
# initial basis, put in |01> state with Sx operator on q0
psi0 = np.zeros((4,1))
psi0[0] = 1
psi0 = np.dot(np.kron(I,Sx),psi0)
print(psi0)
```

    [[ 0.]
     [ 1.]
     [ 0.]
     [ 0.]]

We haven't defined our VQE ansatz yet, but before we do, let's write a function to return the expected value of the Hamiltonian `Hmol` given an `ansatz`, its parameter `theta`, and the initial state `psi0`. This ansatz will eventually be encoded by the quantum circuit.

```python
def expected(theta,ansatz,Hmol,psi0):
    circuit = ansatz(theta[0])
    psi = np.dot(circuit,psi0)
    return np.real(np.dot(psi.conj().T,np.dot(Hmol,psi)))[0,0]
```

With the expectation value in had, we now define an ansatz. In the O'Malley paper, they utilize the Unitary Coupled Cluster (UCC) ansatz, which in this case depends only on a single parameter $\theta$:

$U(\theta) = \mathrm{exp}\left(-i\theta X_0Y_1\right)$

so that a parameterized wave function $|\psi(\theta)\rangle$ for the ground state of H$_2$ is given as

$|\psi(\theta)\rangle = \mathrm{exp}\left(-i\theta X_0Y_1\right)|01\rangle$

and $X_0Y_1$ is the tensor product of the Pauli-X on qubit 0 and Pauli-Y on qubit 1.

Before thinking about how we might represent $U(\theta)$ as a series of quantum gates, let's plug it into the expression

$E(\theta) = \frac{\langle \psi | U^{\dagger}(\theta)\hat{H}_{\mathrm{mol}}U(\theta)|\psi\rangle}{\langle \psi | U^{\dagger}(\theta)U(\theta)|\psi\rangle} = \langle \psi | U^{\dagger}(\theta)\hat{H}_{\mathrm{mol}}U(\theta)|\psi\rangle$

(Note that as long as $\psi$ is normalized and $U$ is unitary, we can ignore the normalization $\langle \psi | U^{\dagger}(\theta)U(\theta)|\psi\rangle$, since it always equals 1.)

Given the `ansatz` and the initial state `psi0`, we can minimize `expected()` using the classical optimizers in the `scipy` package. So straightforwardly plugging in and minimizing yields the lazy result:

```python
from scipy.linalg import expm
from scipy.optimize import minimize

# our UCC ansatz, not yet represented in terms of quantum gates
ansatz = lambda theta: expm(-1j*np.array([theta])*np.kron(Sy,Sx))

# initial guess for theta
theta  = [0.0]
result = minimize(expected,theta,args=(ansatz,Hmol,psi0))
theta  = result.x[0]
val    = result.fun

print("Lazy VQE: ")
print("  [+] theta:  {:+2.8} deg".format(theta))
print("  [+] energy: {:+2.8} Eh".format(val + nuclear_repulsion))
```

    Lazy VQE:
      [+] theta:  -0.11487186 deg
      [+] energy: -1.1456295 Eh

Which equals the result we got from diagonalization of the Hamiltonian. So we know that the UCC ansatz works! But we were lazy and didn't bother to think about how the quantum computer would compute the exponential. So, how do we represent the UCC ansatz in terms of quantum gates acting on the initial qubit state?

### A "real" quantum circuit

So before we saw the UCC ansatz works, but we cheated by keeping it as a matrix exponential. This is not a suitable form for a quantum computer. Let's do better.

According to the O'Malley paper , we can represent $U(\theta)$ for this problem as:

This means that we should change our ansatz to read

```python
# read right-to-left (bottom-to-top?)

ansatz = lambda theta: (np.dot(np.dot(np.kron(-Ry(np.pi/2),Rx(np.pi/2)),
                        np.dot(CNOT10,
                        np.dot(np.kron(I,Rz(theta)),
                               CNOT10))),
                               np.kron(Ry(np.pi/2),-Rx(np.pi/2))))

```

Note that while you read the the circuit diagram left-to-right, when you read the matrix expression above it is better read right-to-left. The right-most matrices are applied to the state first, so that the first gates we apply are -R$_x(\pi/2)$ to qubit-0 and R$_y(\pi/2)$ to qubit-1. Also note that when we apply these two gates, it is simultaneous so the "total" gate is really -R$_y(\pi/2) \otimes $R$_x(\pi/2)$.

```python
theta  = [0.0]
result = minimize(expected,theta,args=(ansatz,Hmol,psi0))
theta  = result.x[0]
val    = result.fun

print("VQE: ")
print("  [+] theta:  {:+2.8} deg".format(theta))
print("  [+] energy: {:+2.8} Eh".format(val + nuclear_repulsion))
```

    VQE:
      [+] theta:  +2.9118489 deg
      [+] energy: -1.1456295 Eh

Which is the correct answer! Since we can now compute the expectation value of our Hamiltonian using quantum gates, we can pass the computed energy to a classical optimizer, which gives new parameters for the quantum gates. When this process is repeated until convergence, we obtain the FCI ground state energy. Also, once we have the optimized wave function parameters, the ground state can be easily reconstructed for additional simulations, etc.

You might have noticed, though, that the above is still not sufficient for a quantum computer. The reason is that although we have represented our wave function with quantum gates, the *measurement* of the expectation value is still poorly defined as a physical operation. Even if you have prepared your qubits to represent a molecular wave function, measuring the expectation value of the Hamiltonian is not simply accomplished physically by applying a "Hamiltonian operation".

An analogy: similar to classical computation, you might want a string, or float, or whatever as the "true" output of your function, but for the computer to compute it for you -- it needs to ultimately be in binary. Same thing for the quantum computer. Our function should ultimately return the energy, but it needs to process this in terms of quantum bits.

### A "real" measurement of the energy

All that is to say that we were cheating again. Experimentally, the "only" measurements we can make are those which probe the final quantum state of the qubits. What we need a way to connect measurements of qubits to an expression for the expectation value of the molecular electronic Hamiltonian.

Put another way, the problem stems back to our defintion of the expected value:

```python
def expected(theta,ansatz,Hmol,psi0):
    circuit = ansatz(theta[0])
    psi = np.dot(circuit,psi0)
    return np.real(np.dot(psi.conj().T,np.dot(Hmol,psi)))[0,0]
```

Simply dotting in `Hmol` with the wave function will not work, because physically we don't have a measuring apparatus for "energy". We can, however, measure the state of each qubit by measuring the spin ($\hat{\mathrm{S}}_z$) of each qubit. We need to reduce the Hamiltonian's expected value into these types of "easy" projective measurements that can be done in the computational basis. These are sums of Pauli measurements.

Now in some respects, we are already halfway there. For a normalized wave function $|\psi'\rangle$:

$$E = \langle \psi'|\hat{H}_\mathrm{mol}|\psi'\rangle$$

and using the definition of our H$_2$ Hamiltonian in the computational basis we have:

$$
\begin{align} E &= \langle \psi'|g_0 \mathbf{I} + g_1 Z_0 + g_2 Z_1 + g_3 Z_0Z_1 + g_4 Y_0Y_1 + g_5 X_0 X_1|\psi'\rangle \\
&= g_0\langle \mathbf{I} \rangle + g_1\langle Z_0 \rangle + g_2 \langle Z_1 \rangle + g_3 \langle Z_0Z_1 \rangle + g_4 \langle Y_0Y_1 \rangle + g_5\langle X_0 X_1 \rangle \\
&= \sum_i g_i \langle \hat{O}_i \rangle
\end{align}
$$

meaning that, given our wave function in the computational basis, if we can compute the expected value of the (products of) Pauli operators, we can relate this to the expected value of the Hamiltonian through the sum given above. This is given in that figure:

Let's go a step further, though. It would be even better if we could relate the above expression to a *single type* of Pauli measurement, that is, measuring the spin of just *one* qubit. Then we don't need to have multiple measurement apparatus.

Thankfully, there is a way to do this. The trick is to apply an additional unitary transformation at the end of the circuit so that, by measuring the spin of the top qubit, we can obtain any Pauli measurement. In our case, that means relating each of the the $\langle \hat{O}_i\rangle$ quantities to the expected value of $\langle Z_1 \otimes \mathbf{I}\rangle$ by some appropriate unitary operator. This is what is meant by the R$_t$ gates in this part of the figure. The R$_t$ are the unitaries we are talking about.

The little measuring gauge means we finally apply the measurement. Because we apply a particular form of the unitaries, we only need to measure the state of qubit-1 like we talked about above, but that's not necessarily the only way to go about it.

You can find a table of some of these transformations [here](https://docs.microsoft.com/en-us/quantum/concepts/pauli-measurements), but here are a few examples:

For example, the simplest case is if you want to measure $Z_1 \otimes \mathbf{I}$. Then you don't have to do anything:

 $\qquad Z_1 \otimes \mathbf{I} = (\mathbf{I} \otimes \mathbf{I})^{\dagger} \otimes (Z_1 \otimes \mathbf{I}) \otimes (\mathbf{I} \otimes \mathbf{I})$

But if you want to measure $Y_1 \otimes \mathbf{I}$, then

$\qquad Y_1 \otimes \mathbf{I} = ({HS^{\dagger}} \otimes \mathbf{I})^{\dagger} \otimes (Z_1 \otimes \mathbf{I}) \otimes ({HS^{\dagger}} \otimes \mathbf{I})$

For $\mathbf{I} \otimes Z_0$, we have to apply the SWAP gate

$\qquad \mathbf{I} \otimes Z_0 = (\mathrm{SWAP})^{\dagger} \otimes (Z_1 \otimes \mathbf{I}) \otimes (\mathrm{SWAP})$

And as a final example, for $X_1Z_0$, (*caveat: we have different qubit ordering compared to the Microsoft documentation, so our CNOT$_{10}$ vs CNOT$_{01}$ definitions are swapped*)

$\qquad X_1 \otimes Z_0 = (\mathrm{CNOT}_{01}(H \otimes \mathbf{I}))^{\dagger} \otimes (Z_1 \otimes \mathbf{I}) \otimes (\mathrm{CNOT}_{01}(H \otimes \mathbf{I}))$

Since you can think of these unitary transformations either acting on the operator, *or* acting on the state, the end result is that by applying the particular transformation and *then* measuring $Z_1$ you can get any Pauli measurement you want.

It might be easier to see by example. Let's see how this plays out for our Hamiltonian.

We will update the function `expected()` to `projective_expected()`, and remove the `Hmol` argument. In our case, this function will include the hard-coded representation of the Hamiltonian, in terms of measuring $Z_1$.

```python
def projective_expected(theta,ansatz,psi0):
    # this will depend on the hard-coded Hamiltonian + coefficients
    circuit = ansatz(theta[0])
    psi = np.dot(circuit,psi0)

    # for 2 qubits, assume we can only take Pauli Sz measurements (Sz \otimes I)
    # we just apply the right unitary for the desired Pauli measurement
    measureZ = lambda U: np.dot(np.conj(U).T,np.dot(np.kron(Sz,I),U))

    energy = 0.0

    # although the paper indexes the hamiltonian left-to-right (0-to-1)
    # qubit-1 is always the top qubit for us, so the tensor pdt changes
    # e.g. compare with the "exact Hamiltonian" we explicitly diagonalized

    # <I1 I0>
    energy += g0 # it is a constant

    # <I1 Sz0>
    U = SWAP
    energy += g1*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    # <Sz1 I0>
    U = np.kron(I,I)
    energy += g2*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    # <Sz1 Sz0>
    U = CNOT01
    energy += g3*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    # <Sx1 Sx0>
    U = np.dot(CNOT01,np.kron(H,H))
    energy += g4*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    # <Sy1 Sy0>
    U = np.dot(CNOT01,np.kron(np.dot(H,S.conj().T),np.dot(H,S.conj().T)))
    energy += g5*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    return np.real(energy)[0,0]

```

With the expectation value now defined in terms of measuring the spin of the zero-th qubit, let's carry out the VQE procedure:

```python
theta  = [0.0]
result = minimize(projective_expected,theta,args=(ansatz,psi0))
theta  = result.x[0]
val    = result.fun

print("VQE: ")
print("  [+] theta:  {:+2.8} deg".format(theta))
print("  [+] energy: {:+2.8} Eh".format(val + nuclear_repulsion))
```

    VQE:
      [+] theta:  +2.9118489 deg
      [+] energy: -1.1456295 Eh

Success! We get the same energy (and theta) as the previous VQE circuit, but now all measurements are related back to the result of measuring the spin of the qubit.

### Appendix: All together now

Here's all of the pieces together in one place

```python
import numpy as np
from scipy.linalg import block_diag
from scipy.optimize import minimize

np.set_printoptions(precision=4,suppress=True)

# Pauli matrices
I  = np.array([[ 1, 0],
               [ 0, 1]])
Sx = np.array([[ 0, 1],
               [ 1, 0]])
Sy = np.array([[ 0,-1j],
               [1j, 0]])
Sz = np.array([[ 1, 0],
               [ 0,-1]])

# Hadamard matrix
H = (1/np.sqrt(2))*np.array([[ 1, 1],
                             [ 1,-1]])

# Phase matrix
S = np.array([[ 1, 0],
              [ 0,1j]])

# single qubit basis states |0> and |1>
q0 = np.array([[1],
               [0]])
q1 = np.array([[0],
               [1]])

# Projection matrices |0><0| and |1><1|
P0  = np.dot(q0,q0.conj().T)
P1  = np.dot(q1,q1.conj().T)

# Rotation matrices as a function of theta, e.g. Rx(theta), etc.
Rx = lambda theta : np.array([[    np.cos(theta/2),-1j*np.sin(theta/2)],
                              [-1j*np.sin(theta/2),    np.cos(theta/2)]])
Ry = lambda theta : np.array([[    np.cos(theta/2),   -np.sin(theta/2)],
                              [    np.sin(theta/2),    np.cos(theta/2)]])
Rz = lambda theta : np.array([[np.exp(-1j*theta/2),                0.0],
                              [                0.0, np.exp(1j*theta/2)]])

# CNOTij, where i is control qubit and j is target qubit
CNOT10 = np.kron(P0,I) + np.kron(P1,Sx) # control -> q1, target -> q0
CNOT01 = np.kron(I,P0) + np.kron(Sx,P1) # control -> q0, target -> q1

SWAP   = block_diag(1,Sx,1)

# See DOI: 10.1103/PhysRevX.6.031007
# Here, we use parameters given for H2 at R=0.75A
g0 = -0.4804
g1 = +0.3435
g2 = -0.4347
g3 = +0.5716
g4 = +0.0910
g5 = +0.0910

nuclear_repulsion = 0.7055696146

Hmol = (g0 * np.kron( I, I) + # g0 * I
        g1 * np.kron( I,Sz) + # g1 * Z0
        g2 * np.kron(Sz, I) + # g2 * Z1
        g3 * np.kron(Sz,Sz) + # g3 * Z0Z1
        g4 * np.kron(Sy,Sy) + # g4 * Y0Y1
        g5 * np.kron(Sx,Sx))  # g5 * X0X1

electronic_energy = np.linalg.eigvalsh(Hmol)[0] # take the lowest value
print("Classical diagonalization: {:+2.8} Eh".format(electronic_energy + nuclear_repulsion))
print("Exact (from G16):          {:+2.8} Eh".format(-1.1457416808))

# initial basis, put in |01> state with Sx operator on q0
psi0 = np.zeros((4,1))
psi0[0] = 1
psi0 = np.dot(np.kron(I,Sx),psi0)

# read right-to-left (bottom-to-top?)
ansatz = lambda theta: (np.dot(np.dot(np.kron(-Ry(np.pi/2),Rx(np.pi/2)),
                        np.dot(CNOT10,
                        np.dot(np.kron(I,Rz(theta)),
                               CNOT10))),
                               np.kron(Ry(np.pi/2),-Rx(np.pi/2))))

def projective_expected(theta,ansatz,psi0):
    # this will depend on the hard-coded Hamiltonian + coefficients
    circuit = ansatz(theta[0])
    psi = np.dot(circuit,psi0)

    # for 2 qubits, assume we can only take Pauli Sz measurements (Sz \otimes I)
    # we just apply the right unitary for the desired Pauli measurement
    measureZ = lambda U: np.dot(np.conj(U).T,np.dot(np.kron(Sz,I),U))

    energy = 0.0

    # although the paper indexes the hamiltonian left-to-right (0-to-1)
    # qubit-1 is always the top qubit for us, so the tensor pdt changes
    # e.g. compare with the "exact Hamiltonian" we explicitly diagonalized

    # <I1 I0>
    energy += g0 # it is a constant

    # <I1 Sz0>
    U = SWAP
    energy += g1*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    # <Sz1 I0>
    U = np.kron(I,I)
    energy += g2*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    # <Sz1 Sz0>
    U = CNOT01
    energy += g3*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    # <Sx1 Sx0>
    U = np.dot(CNOT01,np.kron(H,H))
    energy += g4*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    # <Sy1 Sy0>
    U = np.dot(CNOT01,np.kron(np.dot(H,S.conj().T),np.dot(H,S.conj().T)))
    energy += g5*np.dot(psi.conj().T,np.dot(measureZ(U),psi))

    return np.real(energy)[0,0]

theta  = [0.0]
result = minimize(projective_expected,theta,args=(ansatz,psi0))
theta  = result.x[0]
val    = result.fun

# check it works...
assert np.allclose(val + nuclear_repulsion,-1.1456295)

print("VQE: ")
print("  [+] theta:  {:+2.8} deg".format(theta))
print("  [+] energy: {:+2.8} Eh".format(val + nuclear_repulsion))
```

    Classical diagonalization: -1.1456295 Eh
    Exact (from G16):          -1.1457417 Eh
    VQE:
      [+] theta:  +2.9118489 deg
      [+] energy: -1.1456295 Eh

```python

```


# Iterasi **For** dan **While**

Akan diperkenalkan terlebih dahulu suatu fungsi atau **syntax** yang dapat digunakan untuk melakukan suatu hal secara berulang - ulang. Hal ini yang disebut dengan iterasi dan pada Python penulisan iterasi tersebut adalah sebagai berikut :
```
for dummy variable in range (banyak iterasi):
    action
```
Ada yang unik dengan fungsi **for**, yaitu iterasi dimulai dari 0 dan berhenti pada indeks sebelum banyak iterasi yang ditentukan. Contoh
```
for i in range(5):
```
Maka **i** akan diulang sebanyak 5 kali dengan nilai **i** adalah 0,1,2,3 dan 4 secara berururtan.
dapat pula dibuat pengulangan berdasarkan suatu kelipatan tertentu ataupun mulai dari indeks-indeks tertentu.  Secara bentuk seperti berikut :

```
(start, stop, step)
```
dengan start adalah indeks pertama, stop adalah batas indeks terakhir dan step adalah kelipatan indeks yang akan diambil.

```
for i in range(5):
```

Untuk pengulangan yang tak hingga dapat digunakan fungsi **while**
```
while condition  :
    action (jangan lupa untuk mengubah kondisi agar iterasi berhenti)
```

```python
# Contoh

for i in range(3):
    print(i)
```

    0
    1
    2

```python
for k in range(1,24, 3):
    print(k)
```

    1
    4
    7
    10
    13
    16
    19
    22

```python
cond = True
n = 0
while cond == True:
    if n != 0 :
        n-=1
        print(n)
        cond = False
    else :
        n+=1
        print(n)
```

    1
    0

Misal akan dihitung jumlah deret berikut :
\begin{equation}
\sum_{i = 1}^{10} i = 1+2+3+4+5+6+7+8+9+10 = 55
\end{equation}

```python
sum = 0
for i in range(1,11):
    print(i)
    sum += i
sum
```

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10

    55

# Latihan
Hitunglah jumlah deret berikut :
\begin{equation}
\sum_{i = 1}^{10} i^2
\end{equation}
\begin{equation}
\frac{1}{10}\sum_{n = 1}^{10} (n-3)^2
\end{equation}

\begin{equation}
\sum_{j = 1}^{5} \sum_{i = 1}^{10} ij
\end{equation}

# Tranpose

Operator Transpose adalah operator yang "memindahkan" atau "memutar" matriks yang dinotasikan seperti pangkat. Misalkan matriks $\mathbf{X}$ sebagai berikut
\begin{equation}
\mathbf{X} =
\begin{bmatrix}
x_{11} & x_{12} & \cdots & x_{1n} \\
x_{21} & x_{22} & \cdots & x_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n1} & x_{n2} & \cdots & x_{nn} \\
\end{bmatrix}
\end{equation}
Maka transpose dari matrix $\mathbf{X}$ adalah sebagai berikut :
\begin{equation}
\mathbf{X}^T =
\begin{bmatrix}
x_{11} & x_{21} & \cdots & x_{n1} \\
x_{12} & x_{22} & \cdots & x_{n2} \\
\vdots & \vdots & \ddots & \vdots \\
x_{1n} & x_{2n} & \cdots & x_{nn} \\
\end{bmatrix}
\end{equation}

# Latihan !
Tentukanlah trasnpose untuk matriks berikut
\begin{equation}
\mathbf{A} =
\begin{bmatrix}
1 & 2 \\
3 & 2 \\
2 & 1 \\
5 & 6
\end{bmatrix}
\end{equation}

\begin{equation}
\mathbf{B} =
\begin{bmatrix}
1 & 2 &3\\
3 & 2 & 1\\
\end{bmatrix}
\end{equation}

# Perkalian Matriks
Misalkan kembali $\mathbf{X}$ dan $\mathbf{Y}$ adalah suatu matriks ukuran $n\times n$
\begin{equation*} \mathbf{X} =
\begin{bmatrix}
x_{11} & x_{12} & \cdots & x_{1n} \\
x_{21} & x_{22} & \cdots & x_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n1} & x_{n2} & \cdots & x_{nn} \\
\end{bmatrix} \text{ dan }
\mathbf{Y} =
\begin{bmatrix}
y_{11} & y_{12} & \cdots & y_{1n} \\
y_{21} & y_{22} & \cdots & y_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
y_{n1} & y_{n2} & \cdots & y_{nn} \\
\end{bmatrix}
\end{equation*}

Perkalian matriks memiliki suatu aturan khusus yang perlu dipenuhi seperti berikut :
\begin{equation*}
(n \times m) \cdot (m \times n) = (n \times n)
\end{equation*}
dengan kata lain ukuran baris dari vektor / matriks yang pertama harus sama dengan ukuran kolom dari vektor / matriks. Contoh :
\begin{align*}
(2 \times 3) \cdot (3 \times 2) = (2 \times 2) \\
(3 \times 2) \cdot (2 \times 4) = (3 \times 4)
\end{align*}
Sehingga untuk perkalian

Sehingga diperoleh :
\begin{equation*} \mathbf{X} \cdot \mathbf{Y} =
\begin{bmatrix}
x_{11} & x_{12} & \cdots & x_{1n} \\
x_{21} & x_{22} & \cdots & x_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n1} & x_{n2} & \cdots & x_{nn} \\
\end{bmatrix} \cdot
\begin{bmatrix}
y_{11} & y_{12} & \cdots & y_{1n} \\
y_{21} & y_{22} & \cdots & y_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
y_{n1} & y_{n2} & \cdots & y_{nn} \\
\end{bmatrix} =
\begin{bmatrix}
x_{11}y_{11} + x_{12}y_{21} + \dots + x_{1n}y_{n1} & x_{11}y_{12} + x_{12}y_{22} + \dots +x_{1n}y_{n2} & \cdots & x_{11}y_{1n} + x_{12}y_{2n} + \dots + x_{1n}y_{nn} \\
x_{21}y_{11} + x_{22}y_{21} + \dots + x_{2n}y_{n1} & x_{21}y_{12} + x_{22}y_{22} + \dots + x_{2n}y_{n2} & \cdots & x_{21}y_{1n} + x_{22}y_{2n} + \dots + x_{2n}y_{nn} \\
\vdots & \vdots & \ddots & \vdots \\
x_{n1}y_{11} + x_{n2}y_{21} + \dots + x_{nn}y_{n1} & x_{n1}y_{12} + x_{n2}y_{22} + \dots + x_{nn}y_{n2} & \cdots & x_{n1}y_{1n} + x_{n2}y_{2n} + \dots + x_{nn}y_{nn} \\
\end{bmatrix}
\end{equation*}

## Latihan !!
- Hitunglah perkalian matriks berikut jika diketahui
\begin{equation*} \mathbf{X} =
\begin{bmatrix}
1 & 2 & -2 & 1 \\
2 & 2 & 2 & 1 \\
1 & 2 & 1 & 1 \\
2 & -1 & 1 & 2 \\
\end{bmatrix} \text{ dan }
\mathbf{Y} =
\begin{bmatrix}
1 & 2 & 2 & -1 \\
-2 & 2 & -2 & 1 \\
1 & -2 & 1 & 1 \\
-2 & 1 & 1 & -2 \\
\end{bmatrix}
\end{equation*}
- $\mathbf{X} \cdot \mathbf{Y}$
- $\mathbf{X} \cdot \mathbf{Y}^2$
- $2\mathbf{X} \cdot 3\mathbf{Y}$
- $\mathbf{X}^T \cdot \mathbf{Y}$

# Tambahan
Operator pada matriks lainnya adalah Trace yang dinotasikan dengan $\text{tr}(\cdot)$. Trace adalah jumlah dari elemen diagonal suatu matriks persegi


 INF285 - Computación Científica

 Roots of 1D equations

  [S]cientific [C]omputing [T]eam

 Version: 1.37

## Table of Contents
* [Introduction](#intro)
* [Bisection Method](#bisection)
* [Fixed Point Iteration and Cobweb diagram](#fpi)
* [FPI - example from etxtbook](#fpi-textbook-example)
* [Newton Method](#nm)
* [Wilkinson Polynomial](#wilkinson)
* [Acknowledgements](#acknowledgements)
* [Extra Examples](#extraexamples)

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
%matplotlib inline
from ipywidgets import interact
from ipywidgets import widgets
sym.init_printing()
from scipy import optimize
import pandas as pd
pd.set_option("display.colheader_justify","center")
pd.options.display.float_format = '{:.10f}'.format
from colorama import Fore, Back, Style
# https://pypi.org/project/colorama/
# Fore: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Back: BLACK, RED, GREEN, YELLOW, BLUE, MAGENTA, CYAN, WHITE, RESET.
# Style: DIM, NORMAL, BRIGHT, RESET_ALL
textBold = lambda x: Style.BRIGHT+x+Style.RESET_ALL
textBoldH = lambda x: Style.BRIGHT+Back.YELLOW+x+Style.RESET_ALL
```

# Introduction
[Back to TOC](#toc)

In this document we're going to study how to find roots of 1D equations using numerical methods.
First, let's start with the definition of a root:

Definition: The function $f(x)$ has a root in $x = r$ if $f(r) = 0$.

Example: Let's say we want to solve the equation $r + \log(r) = 3$.
We can re-arrange the equation as follows: $r + \log(r) - 3 = 0$.
Thus, solving the previous equation is equivalent to find the root of $f(x) = x + \log(x) - 3$.
This example shows how we can translate an equation into a root-finding problem!
We will study now several numerical methods to find roots.

We will start by defining a function $f(x)$ using a __lambda__ definition.

```python
f = lambda x: x+np.log(x)-3
```

Notice that we have used the NumPy implementation for the logarithmic function. _**Quick question**: what is the base for this implementation? Is it the natural logarithm or logarithm base 10?_

But before we start working on a numerical implementation, we should always consider in solving the problem algebraically.
This can be done with SymPy, i.e. using symbolic computation.
For instance, we will start by defining a symbolic variable:

```python
# Definition of symbolic variable
x = sym.Symbol('x')
# Defining 'symbolic' function
fsym = lambda x: x+sym.log(x)-3
# Finding the root 'symbolically' and obtaining the only root
r=sym.solve(sym.Eq(fsym(x), 0), x)[0]
print(textBoldH('Root obtained:'),r)
print(textBoldH('Numerical root:'),r.evalf())
```

    Root obtained: LambertW(exp(3))
    Numerical root: 2.20794003156932

**Lamber W function**: https://en.wikipedia.org/wiki/Lambert_W_function

We will now obtain the root 'manually'.
This is not a recommended path but it is useful initially.

```python
def find_root_manually(r=2.0):
    # Defining a vector to evaluate f(x) in a vectorized fashion
    x = np.linspace(1,3,1000)
    # reating the figure
    plt.figure(figsize=(8,8))
    # Plotting the function in a vectorized way
    plt.plot(x,f(x),'b-')
    # Plotting the x-axis.
    # Quick question: Why do we have to multiply 'x' by '0'? What would happen if we only put '0' instead of 'x*0'?
    plt.plot(x,x*0,'r--')
    # Adding the background grid to the plot. We strongly recommend it!
    plt.grid(True)
    # Just adding labels.
    plt.ylabel('$f(x)$',fontsize=16)
    plt.xlabel('$x$',fontsize=16)
    plt.title('$r='+str(r)+',\, f(r)='+str(f(r))+'$',fontsize=16)
    plt.plot(r,f(r),'k.',markersize=20)
    plt.show()
interact(find_root_manually,r=(1,3,1e-3))
```

    interactive(children=(FloatSlider(value=2.0, description='r', max=3.0, min=1.0, step=0.001), Output()), _dom_c…

    <function __main__.find_root_manually(r=2.0)>

# Bisection Method
[Back to TOC](#toc)

The bisection method finds the root of a function $f$.
It requires that:
1. $f$ be a **continuous** function.
2. The interval $[a,b]$, such that $f(a)\cdot f(b) < 0$.

If these 2 conditions are satisfied, it means that there is a value $r$, between $a$ and $b$, for which $f(r) = 0$.
To summarize how this method works, start with the aforementioned interval (checking that there's a root in it), and split it into two smaller intervals $[a,c]$ and $[c,b]$.
Then, check which of the two intervals contains a root.
Keep splitting each "eligible" interval until the algorithm converges or the tolerance is achived.

```python
def bisect(f, a, b, tol=1e-5, maxNumberIterations=100):
    # Evaluating the extreme points of the interval provided
    fa = f(a)
    fb = f(b)
    # Iteration counter.
    i = 0
    # Just checking if the sign is not negative => not root  necessarily
    if np.sign(f(a)*f(b)) >= 0:
        print('f(a)f(b)<0 not satisfied!')
        return None

    # Output table to store the numerical evolution of the algorithm
    output_table = []

    # Main loop: it will iterate until it satisfies one of the two criterias:
    # The tolerance 'tol' is achived or the max number of iterations is reached.
    while ((b-a)/2 > tol) and i<=maxNumberIterations:
        # Obtaining the midpoint of the interval. Quick question: What could happen if a different point is used?
        c = (a+b)/2.
        # Evaluating the mid point
        fc = f(c)
        # Saving the output data
        output_table.append([i, a, c, b, fa, fc, fb, b-a])

        # Did we find the root?
        if fc == 0:
            print('f(c)==0')
            break
        elif np.sign(fa*fc) < 0:
            # This first case consider that the new inetrval is defined by [a,c]
            b = c
            fb = fc
        else:
            # This second case consider that the new interval is defined by [c,b]
            a = c
            fa = fc
        # Increasing the iteration counter
        i += 1

    # Showing final output table
    columns    = ['$i$', '$a_i$', '$c_i$', '$b_i$', '$f(a_i)$', '$f(c_i)$', '$f(b_i)$', '$b_i-a_i$']
    df = pd.DataFrame(data=output_table, columns=columns)
    display(df)

    # Computing the best approximation obtaind for the root, which is the midpoint of the final interval.
    xc = (a+b)/2.
    return xc
```

```python
# Initial example
f1 = lambda x: x+np.log(x)-3
# A different function, notice that x is multiplied to the exponential now and not added, as before.
f2 = lambda x: x*np.exp(x)-3
# This is the introductory example about Fixed Point Iteration
f3 = lambda x: np.cos(x)-x
bisect(f1,1e-10,3) # Recall to change the 'tol'!
```

It's very important to define a concept called **convergence rate**.
This rate shows how fast the convergence of a method is at a specified point.

The convergence rate for the bisection is always 0.5 because this method uses the half of the interval for each iteration.

In this particular case we observe $e_{i+1} \approx \dfrac{e_{i}}{2}$, why? where?

# Fixed Point Iteration and Cobweb diagram
[Back to TOC](#toc)

To learn about the Fixed-Point Iteration we will first learn about the concept of a Fixed Point.

A Fixed Point of a function $g$ is a real number $r$, where $g(r) = r$

The Fixed-Point Iteration is based in the Fixed Point concept and works like this to find the root of a function:

\begin{align*}
    x_{0} &= initial\_guess \\
    x_{i+1} &= g(x_{i})
\end{align*}

To find an equation's root using this method, we'll have to rearrange the equation to make it of the following form $x = g(x)$.
For example, if we want to obtain the root of $f(r)=0$, one could add a zero convenient this way, $f(r)+r=r$, i.e. we add it $r$ on both sides.
This way we have $g(r)=r+f(r)$ and the fixed point iteration could be performed.
In the following example, we'll find the root of $f(x) = x - \cos(x)$ by iterating over the funcion $g(x) = \cos(x)$.

```python
# Just plotting the Cobweb diagram: https://en.wikipedia.org/wiki/Cobweb_plot
def cobweb(x,g=None):
    min_x = np.amin(x)
    max_x = np.amax(x)

    plt.figure(figsize=(10,10))
    ax = plt.axes()
    plt.plot(np.array([min_x,max_x]),np.array([min_x,max_x]),'b-')
    for i in np.arange(x.size-1):
        delta_x = x[i+1]-x[i]
        head_length =  np.abs(delta_x)*0.04
        arrow_length = delta_x-np.sign(delta_x)*head_length
        ax.arrow(x[i], x[i], 0, arrow_length, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')
        ax.arrow(x[i], x[i+1], arrow_length, 0, head_width=1.5*head_length, head_length=head_length, fc='k', ec='k')

    if g!=None:
        y = np.linspace(min_x,max_x,1000)
        plt.plot(y,g(y),'r')

    plt.title('Cobweb diagram')
    plt.grid(True)
    plt.show()

# This code performs the fixed point iteration.
def fpi(g, x0, k, flag_cobweb=False):
    # This is where we store all the approximation,
    # this is technically not needed but we store them because we need them for the cobweb diagram at the end.
    x = np.empty(k+1)
    # Just starting the fixed point iteration from the 'initial guess'
    x[0] = x0
    # Initializing the error in NaN
    error_i = np.nan

    # Output table to store the numerical evolution of the algorithm
    output_table = []

    # Main loop
    for i in range(k):
        # Iteration
        x[i+1] = g(x[i])
        # Storing error from previous iteration
        error_iminus1 = error_i
        # Computing error for current iteration.
        # Notice that from the theory we need to compute e_i=|x_i-r|, i.e. we need the root 'r'
        # but we don't have it, so we approximate it by 'x_{i+1}'.
        error_i = abs(x[i]-x[i+1])
        output_table.append([i,x[i],x[i+1],error_i,error_i/error_iminus1,error_i/(error_iminus1**((1+np.sqrt(5))/2.)),error_i/(error_iminus1**2)])

    # Showing final output table
    columns    = ['$i$', '$x_i$', '$x_{i+1}$', '$e_i$', r'$\frac{e_i}{e_{i-1}}$', r'$\frac{e_i}{e_{i-1}^\alpha}$', r'$\frac{e_i}{e_{i-1}^2}$']
    df = pd.DataFrame(data=output_table, columns=columns)
    display(df)

    # Just showing cobweb if required
    if flag_cobweb:
        cobweb(x,g)
    return x[-1]
```

```python
# First example
g = lambda x: np.cos(x)
# Examples from classnotes
g1 = lambda x: -(3/2)*x+5/2
g2 = lambda x: -(1/2)*x+3/2
fpi(g, 1.1, 20, True)

# Suggestions:
# 1.- A very useful and simple 'limit cicle' example! Try it. Credit: anonymous student from class 20210922.
# fpi(lambda x: -x, 2, 10, True)
# 2.- Try the next example. Why do we see 1.0000 for e_{i+1}/e_i over 90 iterations?
# fpi(g, 1, 100, True)
# 3.- The following fixed-point iteration obtain sqrt(2). Credits: Anonymous student from class 20210921 and 20210922.
# gD = lambda x: (x+2)/(x+1)
# fpi(gD, 1, 10, True)
```

Let's quickly explain the Cobweb Diagram we have here. The blue line is the function $y=x$ and the red is the function $y=g(x)$.
The point in which they meet is $r=g(r)$, i.e. the fixed point.
In this particular example, we start at $y = x = 1.5$ (the top right corner) and then we "jump" **vertically** to $y = \cos(1.5) \approx 0.07$.
After this, we jump **horizontally** to $x = \cos(1.5) \approx 0.07$.
Then, we jump again **vertically** to $y = \cos\left(\cos(1.5)\right) \approx 0.997$ and so on.
See the pattern here? We're just iterating over $x = \cos(x)$, getting closer to the center of the diagram where the fixed point resides, in $x \approx 0.739$.

It's very important to mention that the algorithm will converge only if the rate of convergence $S < 1$, where $S = \left| g'(r) \right|$.
If you want to use this method, you'll have to construct $g(x)$ starting from $f(x)$ accordingly.
In this example, $g(x) = \cos(x) \Rightarrow g'(x) = -\sin(x)$ and $|-\sin(0.739)| \approx 0.67$.
**Quick question:** Do you see the value 0.67 in the previous table?

### Another example. Look at this web page to undertand the context: https://divisbyzero.com/2008/12/18/sharkovskys-theorem/amp/?__twitter_impression=true

```python
# Consider this funtion
g = lambda x: -(3/2)*x**2+(11/2)*x-2
# Here we compute the derivative of it.
gp = lambda x: -3*x+11/2

# We plot now the funcion itself (red), its derivative (magenta) and the function y=x (blue).
# We also plot the values -1 and 1 with green dashed curves.
# This analyis shows that the fixed point, which is the intersection between teh red and blue curves,
# does not generate a convergent fix-point-iteration since the derivative (magenta curve) has a value
# lower then -1 about the fized point.
x=np.linspace(2,3,100)
plt.figure(figsize=(8,8))
plt.plot(x,g(x),'r-',label=r'$g(x)$')
plt.plot(x,x,'b-')
plt.plot(x,gp(x),'m-')
plt.plot(x,gp(x)*0+1,'g--')
plt.plot(x,gp(x)*0-1,'g--')
plt.grid(True)
plt.show()
```

What it is interesting about the previous example is that it generates an interesting limit cicle! In the next cell we evaluate the fixed point with initial guess equal to 1. The iteration oscilates generating the following sequence: 1, 2, 3, 1, 2, 3, .... Which is nice!

```python
fpi(g, 1, 12, True)

# Suggestion, try the following alternative.
# fpi(g, 2.5, 100, True)
```

However, we prefer **convergent** fixed-point-iterations! Here is interesting way to make a non-convergent FPI into a convergent one.

```python
# This is a "palta" hidden in the code! Think about it. Quick question: what is it doing?
a=-1/(1-(-1.72)) # a = -1 / (1 - g'(r)), where does "a" come from?
g2 = lambda x: x+a*(x-g(x))
fpi(g2, 1, 14, True)
```

# FPI - example from textbook
[Back to TOC](#toc)

This example is from the textbook. We are trying to find a root of $f(x)=x^3+x-1$.

```python
# These are the three functions proposed.
g1 = lambda x: 1-x**3
g2 = lambda x: (1-x)**(1/3)
g3 = lambda x: (1+2*x**3)/(1+3*x**2)
# Change the input function to evaluate different functions.
# Are the three functions convergent fixed point iterations?
fpi(g3, 0.75, 10, True)
```

```python
# This is a "hack" to improve the convergence of g2!
a  = -0.6
g4 = lambda x: x+a*(x-g2(x))
fpi(g4, 0.75, 10, True)
# Why does this hack works?
```

Now that we have found the root, let's compute the derivative of each $g(x)$ used previously and understand what exactly was going on.

```python
g1p = lambda x: -3*x**2
g2p = lambda x: -(1/3)*(1-x)**(-2/3)
g3p = lambda x: ((1+3*x**2)*(6*x**2)-(1+2*x**3)*6*x)/((1+3*x**2)**2)
g4p = lambda x: 1+a*(1-g2p(x))
r=0.6823278038280194
print('What is the conclusion then?')
print([g1p(r), g2p(r), g3p(r), g4p(r)])
```

    What is the conclusion then?
    [-1.3967136956303043, -0.7159663452349291, 0.0, -0.029579807140957426]

```python
# Or it may be better to apply the absolute value.
print(np.abs([g1p(r), g2p(r), g3p(r), g4p(r)]))
```

    [1.3967137  0.71596635 0.         0.02957981]

# Newton's Method
[Back to TOC](#toc)

The Newton's method also finds a root of a function $f(x)$ but it requires its derivative, i.e. $f'(x)$.
The algorithm is as follows:
\begin{align*}
    x_0 &= \text{initial guess},\\
    x_{i+1} &= x_i - \dfrac{f(x_i)}{f'(x_i)}.
\end{align*}
For roots with multiplicity equal to 1, Newton's method convergens quadratically. However, when the multiplicity is larger that 1, it will show linear convergence. Fortunately, we can modify Newton's method if we know the multiplicity of the root, say $m$, this is as follows:
\begin{align*}
    x_0 &= \text{initial guess},\\
    x_{i+1} &= x_i - m\,\dfrac{f(x_i)}{f'(x_i)}.
\end{align*}
This modified version will also show quadratic convergence!

```python
def newton_method(f, fp, x0, rel_error=1e-8, m=1, maxNumberIterations=100):
    #Initialization of hybrid error and absolute
    hybrid_error = 100
    error_i = np.inf

    #print('i |     x_i      |     x_{i+1}  |     |x_{i+1}-x_i|      |  e_{i+1}/e_i | e_{i+1}/e_i^2')
    #print('----------------------------------------------------------------------------------------')

    # Output table to store the numerical evolution of the algorithm
    output_table = []

    #Iteration counter
    i = 0
    while (hybrid_error > rel_error and hybrid_error < 1e12 and i<=maxNumberIterations):
        #Newton's iteration
        x1 = x0-m*f(x0)/fp(x0)

        #Checking if root was found
        if f(x1) == 0.0:
            hybrid_error = 0.0
            break

        #Computation of hybrid error
        hybrid_error = abs(x1-x0)/np.max([abs(x1),1e-12])

        #Computation of absolute error
        error_iminus1 = error_i
        error_i = abs(x1-x0)

        # Storing output data
        output_table.append([i,x0,x1,error_i,error_i/error_iminus1,error_i/(error_iminus1**((1+np.sqrt(5))/2.)),error_i/(error_iminus1**2)])

        #Updating solution
        x0 = x1
        #Increasing iteration counter
        i += 1

    # Showing final output table
    columns    = ['$i$', '$x_i$', '$x_{i+1}$', '$e_i$', r'$\frac{e_i}{e_{i-1}}$', r'$\frac{e_i}{e_{i-1}^\alpha}$', r'$\frac{e_i}{e_{i-1}^2}$']
    df = pd.DataFrame(data=output_table, columns=columns)
    display(df)

    #Checking if solution was obtained
    if hybrid_error < rel_error:
        return x1
    elif i>=maxNumberIterations:
        print('Newton''s Method did not converge. Too many iterations!!')
        return None
    else:
        print('Newton''s Method did not converge!')
        return None
```

First example, let's compute a root of $\sin(x)$, near $x_0=3.1$.

```python
# Example funtion
f = lambda x: np.sin(x)
# The derivative of f
fp = lambda x: np.cos(x)
newton_method(f, fp, 3.1,rel_error=1e-15)
```

Now, we will look at the example when Newton's method shows linear convergence.

```python
f = lambda x: x**2
fp = lambda x: 2*x # the derivative of f
newton_method(f, fp, 3.1, rel_error=1e-1, m=1, maxNumberIterations=10)
```

      $i$
      $x_i$
      $x_{i+1}$
      $e_i$
      $\frac{e_i}{e_{i-1}}$
      $\frac{e_i}{e_{i-1}^\alpha}$
      $\frac{e_i}{e_{i-1}^2}$

      0
      0
      3.1000000000
      1.5500000000
      1.5500000000
      0.0000000000
      0.0000000000
      0.0000000000

      1
      1
      1.5500000000
      0.7750000000
      0.7750000000
      0.5000000000
      0.3813629916
      0.3225806452

      2
      2
      0.7750000000
      0.3875000000
      0.3875000000
      0.5000000000
      0.5853091517
      0.6451612903

      3
      3
      0.3875000000
      0.1937500000
      0.1937500000
      0.5000000000
      0.8983220991
      1.2903225806

      4
      4
      0.1937500000
      0.0968750000
      0.0968750000
      0.5000000000
      1.3787288159
      2.5806451613

      5
      5
      0.0968750000
      0.0484375000
      0.0484375000
      0.5000000000
      2.1160485195
      5.1612903226

      6
      6
      0.0484375000
      0.0242187500
      0.0242187500
      0.5000000000
      3.2476737160
      10.3225806452

      7
      7
      0.0242187500
      0.0121093750
      0.0121093750
      0.5000000000
      4.9844719855
      20.6451612903

      8
      8
      0.0121093750
      0.0060546875
      0.0060546875
      0.5000000000
      7.6500791480
      41.2903225806

      9
      9
      0.0060546875
      0.0030273438
      0.0030273438
      0.5000000000
      11.7412057168
      82.5806451613

      10
      10
      0.0030273438
      0.0015136719
      0.0015136719
      0.5000000000
      18.0201941726
      165.1612903226

    Newtons Method did not converge. Too many iterations!!

So, in the previous example Newton's method showed linear convergence.
But, how can we uss its outcome to improve the convergence?
This can fixed by understanding the following facts:
1. Linear convergence definition: $e_{i+1}/e_i=S$
2. Linear convergence exhibit by Newton's method when the root has multiplicity greater than 1: $S=(m-1)/m$

Connecting the two previous two facts we get,
$$e_{i+1}/e_i=(m-1)/m$$.

From the table we obtain that $e_{i+1}/e_i\approx 0.5$, this implies the following equation,
$$0.5=(m-1)/m.$$
Solving for $m$ we get $m=2$.
Knowing this is very useful because we can use it with the Newton's method and recover its quadratic convergence!

# Wilkinson Polynomial
[Back to TOC](#toc)

https://en.wikipedia.org/wiki/Wilkinson%27s_polynomial

**Final question: Why is the root far far away from $16$?**

```python
x = sym.symbols('x', reals=True)
W=1
for i in np.arange(1,21):
        W*=(x-i)
W # Printing W nicely
```

```python
# Expanding the Wilkinson polynomial
We=sym.expand(W)
We
```

```python
# Just computiong the derivative
Wep=sym.diff(We,x)
Wep
```

```python
# Lamdifying the polynomial to be used with sympy
P=sym.lambdify(x,We)
Pp=sym.lambdify(x,Wep)
```

```python
# Using scipy function to compute a root
root = optimize.newton(P,16)
print(root)
```

```python
newton_method(P, Pp, 16.01, rel_error=1e-10, maxNumberIterations=10)
```

      $i$
      $x_i$
      $x_{i+1}$
      $e_i$
      $\frac{e_i}{e_{i-1}}$
      $\frac{e_i}{e_{i-1}^\alpha}$
      $\frac{e_i}{e_{i-1}^2}$

      0
      0
      16.0100000000
      16.0425006915
      0.0325006915
      0.0000000000
      0.0000000000
      0.0000000000

      1
      1
      16.0425006915
      16.0050204647
      0.0374802268
      1.1532132093
      9.5854070309
      35.4827283637

      2
      2
      16.0050204647
      16.0078597186
      0.0028392538
      0.0757533795
      0.5765549711
      2.0211558458

      3
      3
      16.0078597186
      15.9851271041
      0.0227326145
      8.0065452981
      300.2282246866
      2819.9469801818

      4
      4
      15.9851271041
      16.0029892675
      0.0178621634
      0.7857505074
      8.1457523850
      34.5648982597

      5
      5
      16.0029892675
      16.0136315293
      0.0106422618
      0.5957991522
      7.1690902557
      33.3553746866

      6
      6
      16.0136315293
      16.0001980846
      0.0134334447
      1.2622734683
      20.9176811505
      118.6095110764

      7
      7
      16.0001980846
      16.0178167527
      0.0176186681
      1.3115525115
      18.8204280027
      97.6333725295

      8
      8
      16.0178167527
      16.0287977529
      0.0109810002
      0.6232593802
      7.5634001878
      35.3749429376

      9
      9
      16.0287977529
      16.0248062796
      0.0039914733
      0.3634890493
      5.9080101682
      33.1016340099

      10
      10
      16.0248062796
      15.9972551707
      0.0275511088
      6.9024910407
      209.6945048216
      1729.3090772511

    Newtons Method did not converge. Too many iterations!!

# Acknowledgements
[Back to TOC](#toc)

* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. DI UTFSM. March 2016._ v1.1.
* _Update April 2020 - v1.32 - C.Torres_ : Re-ordering the notebook.
* _Update April 2021 - v1.33 - C.Torres_ : Updating format and re-re-ordering the notebook. Adding 'maxNumberIterations' to bisection, fpi and Newton's method. Adding more explanations.
* _Update April 2021 - v1.33 - C.Torres_ : Updating description and solution of 'Proposed classwork'.
* _Update September 2021 - v1.35 - C.Torres_ : Updating and commeting code more.
* _Update September 2021 - v1.36 - C.Torres_ : Updating the way we show the output tables.
* _Update September 2021 - v1.37 - C.Torres_ : Fixing typo suggested by Nicolás Tapia 2021-2. Thanks Nicolás! And removing extra code.

# Extra examples
[Back to TOC](#toc)

## Propose Classwork
1. Build a FPI such that given $a$ computes $\displaystyle \frac{1}{a}$. The constraint is that you can't use a division in the 'final' FPI. Write down your solution below or go and see the [solution](#sol1).

_Hint: I strongly suggest to use Newton's method._

```python
print('Please try to solve it before you see the solution!!!')
```

    Please try to solve it before you see the solution!!!

2. Build an algorithm that computes $\log(x_i)$ for $x_i=0.1*i+0.5$, for $i\in{0,1,2,\dots,10}$. The only special function available is $\exp(x)$, in particular use _np.exp(x)_. You can also use $*$, $÷$, $+$, and $-$. It would be nice to use the result from previous example to replace $÷$.

# In class

Which function shows quadratic convergence? Why?

```python
g1 = lambda x: (4./5.)*x+1./x
g2 = lambda x: x/2.+5./(2*x)
g3 = lambda x: (x+5.)/(x+1)
fpi(g1, 3.0, 10, True)
```

### Building a FPI to compute the cubic root of 7

```python
# What is 'a'? Can we find another 'a'?
a = -3*(1.7**2)
print(a)
```

    -8.669999999999998

```python
f = lambda x: x**3-7
g =  lambda x: f(x)/a+x
r=fpi(g, 1.7, 14, True)
print(f(r))
```

### Playing with some roots

The following example proposed a particular function $f(x)$.
The idea here is first obtain an initial guess for applying the Newton's method from the plot in semilogy scale.
The plot of $f(x)$ (blue) shows that there seems to be 2 roots in the interval plotted.
Now, the plot of $f'(x)$ (magenta) indicates that the derivative may also have a 0 together with a root, this means that the multiplicity of that root may be higher than 1. **Do you see it?**

```python
f = lambda x: 8*x**4-12*x**3+6*x**2-x
fp = lambda x: 32*x**3-36*x**2+12*x-1

x = np.linspace(-1,1,10000)
plt.figure(figsize=(10,10))

plt.title('What are we seeing with the semiloigy plot? Is this function differentiable?')
plt.semilogy(x,np.abs(f(x)),'b-',label=r'$|f(x)|$')
plt.semilogy(x,np.abs(fp(x)),'m-',label=r'$|fp(x)|$')
plt.grid()
plt.legend()
plt.xlabel(r'$x$',fontsize=16)
plt.show()
```

```python
r=newton_method(f, fp, 0.4, rel_error=1e-8, m=1)
print([r,f(r)])
# Is this showing quadratic convergence? If not, can you fix it?
```

      $i$
      $x_i$
      $x_{i+1}$
      $e_i$
      $\frac{e_i}{e_{i-1}}$
      $\frac{e_i}{e_{i-1}^\alpha}$
      $\frac{e_i}{e_{i-1}^2}$

      0
      0
      0.4000000000
      0.4363636364
      0.0363636364
      0.0000000000
      0.0000000000
      0.0000000000

      1
      1
      0.4363636364
      0.4586595886
      0.0222959522
      0.6131386861
      4.7546137810
      16.8613138686

      2
      2
      0.4586595886
      0.4728665654
      0.0142069768
      0.6371998210
      6.6854108733
      28.5791705436

      3
      3
      0.4728665654
      0.4820874099
      0.0092208445
      0.6490363622
      8.9967528627
      45.6843403691

      4
      4
      0.4820874099
      0.4881331524
      0.0060457425
      0.6556603932
      11.8718891457
      71.1063278311

      5
      5
      0.4881331524
      0.4921210847
      0.0039879323
      0.6596265522
      15.5037819584
      109.1059613639

      6
      6
      0.4921210847
      0.4947614808
      0.0026403961
      0.6620965214
      20.1252241907
      166.0250150478

      7
      7
      0.4947614808
      0.4965138385
      0.0017523577
      0.6636722755
      26.0285404605
      251.3533000745

      8
      8
      0.4965138385
      0.4976786184
      0.0011647799
      0.6646930088
      33.5858180781
      379.3135449254

      9
      9
      0.4976786184
      0.4984536173
      0.0007749988
      0.6653607474
      43.2731170072
      571.2330225492

      10
      10
      0.4984536173
      0.4989696118
      0.0005159945
      0.6658003716
      55.7008361948
      859.0985504567

      11
      11
      0.4989696118
      0.4993133111
      0.0003436993
      0.6660910430
      71.6523877204
      1290.8878511100

      12
      12
      0.4993133111
      0.4995423124
      0.0002290013
      0.6662837900
      92.1337096817
      1938.5659202606

      13
      13
      0.4995423124
      0.4996949214
      0.0001526090
      0.6664112926
      118.4366832979
      2910.0766376211

      14
      14
      0.4996949214
      0.4997966348
      0.0001017134
      0.6664965150
      152.2211476726
      4367.3461398087

      15
      15
      0.4997966348
      0.4998644327
      0.0000677979
      0.6665584076
      195.6204335210
      6553.3001358038

      16
      16
      0.4998644327
      0.4999096253
      0.0000451926
      0.6665778509
      251.3627148814
      9831.8335188516

      17
      17
      0.4999096253
      0.4999397542
      0.0000301289
      0.6666774378
      323.0218359853
      14751.9189004836

      18
      18
      0.4999397542
      0.4999598336
      0.0000200794
      0.6664506601
      414.8682245216
      22119.9939244004

      19
      19
      0.4999598336
      0.4999732282
      0.0000133946
      0.6670815682
      533.6278356649
      33222.1656531949

      20
      20
      0.4999732282
      0.4999821551
      0.0000089268
      0.6664505480
      684.6850645121
      49755.1439166074

      21
      21
      0.4999821551
      0.4999881259
      0.0000059708
      0.6688577677
      883.0272572730
      74926.5788229439

      22
      22
      0.4999881259
      0.4999921287
      0.0000040028
      0.6704027562
      1134.8168859231
      112280.4498807474

      23
      23
      0.4999921287
      0.4999947419
      0.0000026133
      0.6528511548
      1414.9287092325
      163097.2875577784

      24
      24
      0.4999947419
      0.4999967498
      0.0000020079
      0.7683392122
      2167.3273576456
      294016.2831700209

    [0.49999718771149326, 0.0]

## Solutions
Problem: Build a FPI such that given $a$ computes $\displaystyle \frac{1}{a}$

```python
# We are finding the 1/a
# Solution code:
a = 2.1
g = lambda x: 2*x-a*x**2
gp = lambda x: 2-2*a*x
r=fpi(g, 0.7, 7, flag_cobweb=False)
print('Reciprocal found :',r)
print('Reciprocal computed explicitly: ', 1/a)
# Are we seeing quadratic convergence?
```

      $i$
      $x_i$
      $x_{i+1}$
      $e_i$
      $\frac{e_i}{e_{i-1}}$
      $\frac{e_i}{e_{i-1}^\alpha}$
      $\frac{e_i}{e_{i-1}^2}$

      0
      0
      0.7000000000
      0.3710000000
      0.3290000000
      NaN
      NaN
      NaN

      1
      1
      0.3710000000
      0.4529539000
      0.0819539000
      0.2491000000
      0.4951799738
      0.7571428571

      2
      2
      0.4529539000
      0.4750566054
      0.0221027054
      0.2696968100
      1.2656881308
      3.2908355795

      3
      3
      0.4750566054
      0.4761877763
      0.0011311709
      0.0511779387
      0.5398483531
      2.3154603813

      4
      4
      0.4761877763
      0.4761904762
      0.0000026999
      0.0023867984
      0.1580649031
      2.1100246103

      5
      5
      0.4761904762
      0.4761904762
      0.0000000000
      0.0000056698
      0.0156742306
      2.1000206476

      6
      6
      0.4761904762
      0.4761904762
      0.0000000000
      0.0000000000
      0.0000000000
      0.0000000000

    Reciprocal found : 0.47619047619047616
    Reciprocal computed explicitly:  0.47619047619047616

### What is this plot telling us?

This plots shows that, even if we don't know the exact value of $g'(r)$, we can determine if the FPI will convergan by looking at the plot.
In this plot we observe that when plotting $g'(x)$ (magenta), we can determine that the value of $|g'(r)|$ will be less than 1 since it is between the black lines, that are located at $y=-1$ and $y=1$.

```python
xx=np.linspace(0.2,0.8,1000)
plt.figure(figsize=(10,10))
plt.plot(xx,g(xx),'r-',label=r'$g(x)$')
plt.plot(xx,gp(xx),'m-',label=r'$gp(x)$')
plt.plot(xx,xx,'b-',label=r'$x$')
plt.plot(xx,0*xx+1,'k--')
plt.plot(xx,0*xx-1,'k--')
plt.legend(loc='best')
plt.grid()
plt.show()
```

```python

```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
from sympy import symbols, sympify, latex, integrate, solve, solveset, Matrix, expand, factor, primitive, simplify, factor_list
from sympy.parsing.sympy_parser import parse_expr
```

```python
M = Matrix(np.reshape(np.random.random(64), (8,8)) + np.eye(8))
```

```python
M
```

    Matrix([
    [  1.72938010955506,  0.108998619445437, 0.617419413857536,  0.672090300210331, 0.606753741022949, 0.941950391186882, 0.329353672325283, 0.330135340967146],
    [ 0.544003091846511,    1.4571834758357, 0.166588963611674,   0.77871894307536,  0.92759642632636, 0.723380109363728, 0.966571852092427, 0.358684859729231],
    [0.0263348345077091,  0.205412256369131,  1.63480353971179,  0.727417671646605, 0.750735973013235, 0.718917836174743, 0.271872823879635, 0.104501570503404],
    [ 0.479093567931866,  0.612751655750204, 0.220756182952718,    1.2271972517674, 0.571749999543387, 0.745256036655373, 0.436574672095169, 0.787042679860924],
    [ 0.267498882107831,  0.234860551854682, 0.266755446926459,  0.822371203165733,   1.0167665761371, 0.311663043557685, 0.498502741247374, 0.483679012101594],
    [ 0.131011410281342,  0.934240039468298, 0.360473891048842,  0.949236090723573, 0.666398870144586,   1.3850623124344, 0.761510580786222, 0.918311992415843],
    [ 0.779835646960045,  0.206545906782152,  0.31767946760615,  0.993901276871165, 0.621777728829399, 0.120687581485802,  1.97616730627295,  0.73210086488975],
    [  0.34195261826588, 0.0933633873347242, 0.500540923752915, 0.0894789341134482, 0.582565335979311, 0.032149637056864, 0.636587579063847,  1.60563751156564]])

```python
plt.figure(figsize=(8,8))
M = np.array(M).astype('float')
im = plt.imshow(M, cmap=cm.hot)
colorbar(im, shrink=0.8, aspect=8)
```

```python

```


## Neural Networks

Deep Learning is based on artificial neural networks which have been around in some form since the late 1950s. The networks are built from individual parts approximating neurons, typically called units or simply "neurons." Each unit has some number of weighted inputs. These weighted inputs are summed together (a linear combination) then passed through an activation function to get the unit's output.

Mathematically this looks like:

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i +b \right)
\end{align}
$$

With vectors this is the dot/inner product of two vectors:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

## Tensors

It turns out neural network computations are just a bunch of linear algebra operations on *tensors*, a generalization of matrices. A vector is a 1-dimensional tensor, a matrix is a 2-dimensional tensor, an array with three indices is a 3-dimensional tensor (RGB color images for example). The fundamental data structure for neural networks are tensors and PyTorch (as well as pretty much every other deep learning framework) is built around tensors.

With the basics covered, it's time to explore how we can use PyTorch to build a simple neural network.

```python
# First, import PyTorch
import torch
```

```python
def activation(x):
    """ Sigmoid activation function

        Arguments
        ---------
        x: torch.Tensor
    """
    return 1/(1+torch.exp(-x))
```

```python
### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 3 random normal variables
features = torch.randn((1, 5))
# True weights for our data, random normal variables again
weights = torch.randn_like(features)
# and a true bias term
bias = torch.randn((1, 1))
```

```python
activation((features * weights).sum() + bias)
```

    tensor([[0.1595]])

```python
## Calculate the output of this network using matrix multiplication
weights = weights.view(5,1)
activation(torch.mm(features,weights) + bias)
```

    tensor([[0.1595]])

### Stack them up!

That's how you can calculate the output for a single neuron. The real power of this algorithm happens when you start stacking these individual units into layers and stacks of layers, into a network of neurons. The output of one layer of neurons becomes the input for the next layer. With multiple input units and output units, we now need to express the weights as a matrix.

The first layer shown on the bottom here are the inputs, understandably called the **input layer**. The middle layer is called the **hidden layer**, and the final layer (on the right) is the **output layer**. We can express this network mathematically with matrices again and use matrix multiplication to get linear combinations for each unit in one operation. For example, the hidden layer ($h_1$ and $h_2$ here) can be calculated

$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

```python
### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 3 random normal variables
features = torch.randn((1, 3))

# Define the size of each layer in our network
n_input = features.shape[1]     # Number of input units, must match number of input features
n_hidden = 2                    # Number of hidden units
n_output = 1                    # Number of output units

# Weights for inputs to hidden layer
W1 = torch.randn(n_input, n_hidden)
# Weights for hidden layer to output layer
W2 = torch.randn(n_hidden, n_output)

# and bias terms for hidden and output layers
B1 = torch.randn((1, n_hidden))
B2 = torch.randn((1, n_output))
```

```python
## Your solution here
l1 = activation(torch.mm(features,W1) + B1)
activation(torch.mm(l1,W2)+B2)
```

    tensor([[0.3171]])

## Numpy to Torch and back

Special bonus section! PyTorch has a great feature for converting between Numpy arrays and Torch tensors. To create a tensor from a Numpy array, use `torch.from_numpy()`. To convert a tensor to a Numpy array, use the `.numpy()` method.

```python
import numpy as np
np.set_printoptions(precision=8)
a = np.random.rand(4,3)
a
```

    array([[0.83207082, 0.98393234, 0.3379819 ],
           [0.13931795, 0.13878253, 0.40786441],
           [0.6300598 , 0.26411612, 0.64072385],
           [0.54986631, 0.85552863, 0.52448797]])

```python
torch.set_printoptions(precision=8)
b = torch.from_numpy(a)
b
```

    tensor([[0.83207082, 0.98393234, 0.33798190],
            [0.13931795, 0.13878253, 0.40786441],
            [0.63005980, 0.26411612, 0.64072385],
            [0.54986631, 0.85552863, 0.52448797]], dtype=torch.float64)

```python
b.numpy()
```

    array([[0.83207082, 0.98393234, 0.3379819 ],
           [0.13931795, 0.13878253, 0.40786441],
           [0.6300598 , 0.26411612, 0.64072385],
           [0.54986631, 0.85552863, 0.52448797]])

The memory is shared between the Numpy array and Torch tensor, so if you change the values in-place of one object, the other will change as well.

```python
# Multiply PyTorch Tensor by 2, in place
b.mul_(2)
```

    tensor([[1.66414164, 1.96786468, 0.67596381],
            [0.27863590, 0.27756506, 0.81572882],
            [1.26011959, 0.52823223, 1.28144770],
            [1.09973262, 1.71105725, 1.04897593]], dtype=torch.float64)

```python
# Numpy array matches new values from Tensor
a
```

    array([[1.66414164, 1.96786468, 0.67596381],
           [0.2786359 , 0.27756506, 0.81572882],
           [1.26011959, 0.52823223, 1.2814477 ],
           [1.09973262, 1.71105725, 1.04897593]])

```python
# inplace operation done on tensors can also change the origin numpy array
```

intro-to-pytorch/.ipynb_checkpoints/Part 1 - Tensors in PyTorch (Exercises)-checkpoint.ipynb


```python
# INPUTS
input_d = 11.1
```

```python
# Setup
from IPython.display import Markdown, Latex, Math
import sympy

```

```python
# Symbol setup
d = sympy.Symbol('d')

# Math formatting. Use Math() for inline math, i.e. equivalent of $math$ for in Latex.
d_latex = Math("d={}".format(input_d))
```

## Inputs

Circle diameter:

{{ d_latex }}

## Area

```python
# Area formula
area_formula = sympy.sympify("pi*(d/2)**2")

# Substitute values without simplifying
with sympy.evaluate(False):
    area_formula_sub = area_formula.subs(d, input_d)

# Compute answer
area = area_formula_sub.evalf()

# Output the result as an aligned equation.
# The asterisk in `\begin{align*}` tells Latex to not add an equation number on the right hand side.
# Note that str.format() uses curly braces for formatting, so any curly braces that are not part
# of a string replacement must be doubled. Hence `\begin{{align*}}` instead of `\begin{align*}`.
area_block = Latex(r"""
\begin{{align*}}
A & = {area_formula} \\
  & = {area_formula_sub} \\
  & = {area_value}
\end{{align*}}
""".format(area_formula=sympy.latex(area_formula),
           area_formula_sub=sympy.latex(area_formula_sub),
           area_value=sympy.latex(area)))
```

The area of a circle with diameter {{ d_latex }} is:

{{ area_block }}

## Circumference

```python
# Circumference formula
circ_formula = sympy.sympify("pi*d")

# Substitute values without simplifying
with sympy.evaluate(False):
    circ_formula_sub = circ_formula.subs(d, input_d)

# Compute answer
circ = circ_formula_sub.evalf()

# Simple one-line equation. In regular Markdown this can be done with $$ equation $$ syntax. However,
# Latex formatting currently cannot be done inline in a Markdown cell because of evaluation order issues when
# Latex and Python-markdown Jupyter extensions are used together.
circ_formula_latex = Markdown("$$ C = " + sympy.latex(circ_formula) +
                              " = " + sympy.latex(circ_formula_sub) +
                              " = " + sympy.latex(circ) + "$$")

```

The circumference of a circle with diameter {{ d_latex }} is:

{{ circ_formula_latex }}

```python

```


# Neural Nets v2
`nn_v2`

Should do [Working efficiently with jupyter lab](https://florianwilhelm.info/2018/11/working_efficiently_with_jupyter_lab/)

When this was a notebook with integrated tests, we did: \
`
%load_ext autoreload
%autoreload 2
%matplotlib widget
#%matplotlib inline`

```python
# import Importing_Notebooks
import numpy as np
```

A network built of components which:
1. accept an ordered set of reals (we'll use `numpy.array`, and  call them vectors) at the input port and produce another at the output port - this is forward propagation. ${\displaystyle f\colon \mathbf {R} ^{n}\to \mathbf {R} ^{m}}$
1. accept an ordered set of reals at the output port, representing the gradient of the loss function at the output, and produce the gradient of the loss function at the input port - this is back propagation, aka backprop. ${\displaystyle b\colon \mathbf {R} ^{m}\to \mathbf {R} ^{n}}$
1. from the gradient of the loss function at the output, calculate the partial of the loss function w.r.t the internal parameters ${\displaystyle \frac{\partial E}{\partial w} }$
1. accept a scalar $\eta$ to control the adjustment of internal parameters. _Or is this effected by scaling the loss gradient before passing??_
1. update internal parameters ${\displaystyle w \leftarrow w - \eta \frac{\partial E}{\partial w} }$

```python
class Layer:
    def __init__(self):
        pass

    def __call__(self, x):
        """Computes response to input"""
        raise NotImplementedError

    def backprop(self, output_delE):
        """Uses output error gradient to adjust internal parameters, and returns gradient of error at input"""
        raise NotImplementedError

    def state_vector(self):
        """Provide the layer's learnable state as a vector"""
        raise NotImplementedError
```

A network built of a cascade of layers:

```python
class Network:
    def __init__(self):
        self.layers = []
        self.eta = 0.1 #FIXME

    def extend(self, net):
        self.layers.append(net)

    def __call__(self, input):
        v = input
        for net in self.layers:
            v = net(v)
        return v

    def learn(self, facts):
        for (x, expected) in facts:
            y = self(x)
            e = y - expected
            loss = float(e.dot(e.T))/2.0
            egrad = e * self.eta
            for net in reversed(self.layers):
                egrad = net.backprop(egrad)
        return loss

    def state_vector(self):
        """Provide the network's learnable state as a vector"""
        return np.concatenate([layer.state_vector() for layer in self.layers])
```

___

## Useful Layers

### Identify

```python
class IdentityLayer(Layer):
    def __call__(self, x):
        return x

    def backprop(self, output_delE):
        return output_delE

    def state_vector(self):
        return np.array([])
```

### Affine
A layer that does an [affine transformation](https://mathworld.wolfram.com/AffineTransformation.html) aka affinity, which is the classic fully-connected layer with output offsets.

$$ \mathbf{M} \mathbf{x} + \mathbf{b} = \mathbf{y} $$
where
$$
\mathbf{x} = \sum_{j=1}^{n} x_j \mathbf{\hat{x}}_j \\
\mathbf{b} = \sum_{i=1}^{m} b_i \mathbf{\hat{y}}_i \\
\mathbf{y} = \sum_{i=1}^{m} y_i \mathbf{\hat{y}}_i
$$
and $\mathbf{M}$ can be written
$$
\begin{bmatrix}
    m_{1,1} & \dots & m_{1,n} \\
    \vdots & \ddots & \vdots \\
    m_{m,1} & \dots & m_{m,n}
\end{bmatrix} \\
$$

#### Error gradient back-propagation
$$
\begin{align}
 \frac{\partial loss}{\partial\mathbf{x}}
  &= \frac{\partial loss}{\partial\mathbf{y}} \frac{\partial\mathbf{y}}{\partial\mathbf{x}} \\
  &= \mathbf{M}^\mathsf{T}\frac{\partial loss}{\partial\mathbf{y}}
\end{align}
$$

#### Parameter adjustment
$$
 \frac{\partial loss}{\partial\mathbf{M}}
 = \frac{\partial loss}{\partial\mathbf{y}} \frac{\partial\mathbf{y}}{\partial\mathbf{M}}
 = \frac{\partial loss}{\partial\mathbf{y}} \mathbf{x} \\
 \frac{\partial loss}{\partial\mathbf{b}}
 = \frac{\partial loss}{\partial\mathbf{y}} \frac{\partial\mathbf{y}}{\partial\mathbf{b}}
 = \frac{\partial loss}{\partial\mathbf{y}}
$$

```python
def column_vecify(m): #FIXME: find the numpy way
    return m.reshape((len(m),1))

class AffinityLayer(Layer):
    """An affine transformation, which is the classic fully-connected layer with offsets"""
    def __init__(self, n, m):
        self.M = np.empty((m, n))
        self.b = np.empty((m, 1))
        self.randomize()

    def randomize(self):
        self.M[:] = np.random.randn(*self.M.shape)
        self.b[:] = np.random.randn(*self.b.shape)

    def __call__(self, x):
        self.input = x
        self.output = self.M @ x + self.b
        return self.output

    def backprop(self, output_delE):
        input_delE = self.M.T @ output_delE
        self.M -= np.einsum('ik,jk', output_delE, self.input) \
            if len(output_delE.shape) == 2 \
            else np.outer(output_delE, self.input)
        self.b -= column_vecify(np.sum(output_delE,axis=1)) \
            if len(output_delE.shape) == 2 \
            else output_delE
        return input_delE

    def state_vector(self):
        return np.concatenate((self.M.ravel(), self.b.ravel()))
```

### Map
Maps a scalar function on the inputs, for e.g. activation layers.

```python
class MapLayer(Layer):
    """Map a scalar function on the input taken element-wise"""
    def __init__(self, fun, dfundx):
        self.vfun = np.vectorize(fun)
        self.vdfundx = np.vectorize(dfundx)

    def __call__(self, x):
        self.input = x
        return self.vfun(x)

    def backprop(self, output_delE):
        input_delE = self.vdfundx(self.input) * output_delE
        return input_delE

    def state_vector(self):
        return np.array([])
```

___

To produce an importable `nn_v2.py`:
1. Save this notebook
1. Uncomment the `jupyter nbconvert` line below
1. Execute it.
1. Comment out the convert again
1. Save the notebook again in that form

```python
### !jupyter nbconvert --to script nn_v2.ipynb
```

    [NbConvertApp] Converting notebook nn_v2.ipynb to script
    [NbConvertApp] Writing 6390 bytes to nn_v2.py

```python

```