# Elementary arithmetic operations
In this tutorial, we are going to provide a construction of quantum networks effecting basic arithmetic operations, covering from addition to modular exponentiation, providing some executable examples using the simulator and five qubit device (ibmqx2).

\begin{align}
Plain\; addition \rightarrow Modular\; addition \rightarrow Modular\; multiplication \rightarrow Modular\; exponentiation
\end{align}

### Contributors

Carlos Bravo Prieto

## Introduction
Nowadays, there is no known efficient classical algorithm for factoring. In the past years, the mathematician and MIT professor Peter Shor discovered an efficient algorithm for factoring large numbers using quantum effects, showing the potential power of quantum computation.
There is a lot of interest in an efficient factoring algorithm, because most of the modern encryption is based on the impossibility to factorize large numbers. Therefore, Shor's algorithm could represent the fall of the actual cryptography.

Shor's algorithm uses the quantum fourier transform to find the order $r$ of a randomly chosen number $x$ in the multiplicative group. This is done by exponentiating $x$ modulo $N$. Here, we focus in the quantum modular exponentiation which seems to be the most difficult part. Although the quantum modular exponentiation does not represent any speed up respect their classical analog, it has to be done in order to implement the quantum Shor's algorithm.

## Plain addition
Our first step is to build the most basic operation, the plain addition, which can be written as

\begin{align}
|a, b \rangle \, \rightarrow \, |a, a+b \rangle \,.
\end{align}

To prevent overflows, if $a$ and $b$ are encoded in $n$ qubits, the second register must be of size $n+1$. We will provisionally write the carries of the addition in a temporary register of size $n-1$ initially in state $|0\rangle$. The operation of the plain addition can be understood as:
1. First we compute the most significant bit of the result $a + b$. We have to compute all the carries $c_i$ through
the relation $c_i$ $\leftarrow$ $a_i$ AND $b_i$ AND $c_{i−1}$, where $a_i$, $b_i$ and $c_i$ represent the $i$th qubit of the first, second and temporary register respectively.
2. Finally we reverse all these operations, except for the last one which computed the leading bit of the
result. This way we can reuse the same temporary register if we require repeated additions. Meanwhile we are doing the resetting process, the other qubits of the result are computed through the relation $b_i$ $\leftarrow$ $a_i$ XOR $b_i$ XOR $c_{i−1}$.

The total number of qubits needed is $3n$.

With this figure we seek a generalization of the algorithm, but the first carry qubit $c_0$ is not really needed, because the value will always be $0$.

### Implementation in ibmqx2. Example 1.
We are going to start with the easiest example, the implementation of the plain addition of two one-qubit numbers $a$ and $b$. This program can be executed in ibmqx2 (5 qubits).

```python
# checking the version of PYTHON; only support > 3.5
import sys
if sys.version_info < (3,5):
    raise Exception('Please use Python version 3.5 or greater.')

# importing QISKit
from qiskit import QuantumProgram
#import Qconfig

# import basic plotting tools
from qiskit.tools.visualization import plot_histogram
from qiskit.tools.qcvv.tomography import marginal_counts

# create a QuantumProgram object instance.
q_program = QuantumProgram()
#q_program.set_api(Qconfig.APItoken, Qconfig.config["url"]) # set the APIToken and API url

# backend
#backend = 'ibmqx2'
backend = 'local_qasm_simulator'
```

First let's define the plain addition algorithm for two one-qubit numbers $a$ and $b$. We implement the quantum plain addition algorithm based on the previous figure. We also define a function that creates those initial states (assuming that our input in QISKIT code is integer). We have to build the Toffoli gate from the elementary gates of the quantum computer.

```python
# quantum plain addition algorithm for 1-qubit numbers
def addition_1bit(circuit, q):
    circuit.h(q[2])
    circuit.cx(q[1], q[2])
    circuit.tdg(q[2])
    circuit.cx(q[0], q[2])
    circuit.t(q[2])
    circuit.cx(q[1], q[2])
    circuit.tdg(q[2])
    circuit.cx(q[0], q[2])
    circuit.t(q[2])
    circuit.h(q[2])
    circuit.t(q[1])
    circuit.cx(q[0], q[1])
    circuit.t(q[0])
    circuit.tdg(q[1])

# n-qubit number input state
def number_state(circuit, q, a, b):
    if a == 1:
        circuit.x(q[0]) # q[0] contains the value of a
    if b == 1:
        circuit.x(q[1]) # q[1] contain the value of b
```

Let's now implement the plain addition of two one-qubit numbers $a+b$, for instance $1+1$, that should return $2 \,(10)$.

In the Quantum Experience composer this circuit would look like:

```python
# we define the values (0 or 1)
a = 1
b = 1

# one single quantum register which contains 'a' (1 qubit) and 'b' (2 qubits)
q = q_program.create_quantum_register("q", 3) # 3 qubits
# clasical register
c = q_program.create_classical_register("cr", 2) # 2 bits

# quantum circuit involving the quantum register and the classical register
add1bit_circuit = q_program.create_circuit("add", [q] ,[c])

# create the state containing a and b
number_state(add1bit_circuit, q, a, b)

# addition
addition_1bit(add1bit_circuit, q)

# measurements to see the result, which has been written in b (q[1]q[2])
add1bit_circuit.measure(q[1], c[0])
add1bit_circuit.measure(q[2], c[1])

# compile and execute the quantum program in the backend
result = q_program.execute(["add"], backend=backend, shots=1024, max_credits=3, wait=10, timeout=999999)

# show the results
print(result)
print(result.get_data("add"))
counts = marginal_counts(result.get_counts("add"), [0, 1])
plot_histogram(counts)
print("Backend:", backend)
print("Highest probability outcome: {}".format(int(max(counts, key = lambda x: counts[x]).replace(" ", ""), 2)))
```

We see that the highest probability outcome is $2 \;(10)$ when we execute the code on ibmqx2.

### Implementation in the quantum simulator. Example 2.
The second example is the implementation of the plain addition of two $n$-qubit numbers $a$ and $b$. This program can be executed in the quantum simulator.

```python
# checking the version of PYTHON; only support > 3.5
import sys
if sys.version_info < (3,5):
    raise Exception('Please use Python version 3.5 or greater.')

# importing QISKit
from qiskit import QuantumProgram

# import basic plotting tools
from qiskit.tools.visualization import plot_histogram

# create a QuantumProgram object instance.
q_program = QuantumProgram()

# backend
backend = 'local_qasm_simulator'
```

First let's define different modules that we are going to use repeatedly, as well as the plain addition algorithm for two $n$-qubit numbers.

```python
def carry(circuit, q0, q1, q2, q3):
    "carry module"
    circuit.ccx(q1, q2, q3)
    circuit.cx(q1, q2)
    circuit.ccx(q0, q2, q3)

def carry_inv(circuit, q0, q1, q2, q3):
    "carry module but running backwards"
    circuit.ccx(q0, q2, q3)
    circuit.cx(q1, q2)
    circuit.ccx(q1, q2, q3)

def summation(circuit, q0, q1, q2):
    "summation module"
    circuit.cx(q1, q2)
    circuit.cx(q0, q2)

# quantum plain addition algorithm for n-qubit numbers
def addition_nbit(circuit, qa, qb, qcar, n):
    if n == 1:
        circuit.ccx(qa[0], qb[0], qb[1])
        circuit.cx(qa[0], qb[0])
    else:
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])
        for i in range(n-2):
            carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1])
        carry(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n])
        circuit.cx(qa[n-1], qb[n-1])
        for i in range(n-1, 1, -1):
            summation(circuit, qcar[i-1], qa[i], qb[i])
            carry_inv(circuit, qcar[i-2], qa[i-1], qb[i-1], qcar[i-1])
        summation(circuit, qcar[0], qa[1], qb[1])
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])
```

Now we are going to define a function that creates the states containing $a$ and $b$.

```python
# n-qubit number input state
def number_state(circuit, q, x, n):
    # integer to binary
    x = "{0:b}".format(x)
    x = x.zfill(n)
    # creating the state
    for i in range(n):
        if int(x[n-1-i]) == 1:
            circuit.x(q[i])
```

Let's now implement the plain addition of two numbers $a+b$, for instance $9+14$, that should return $23$ $(10111)$:

```python
# we define the values
a = 9
b = 14

# computing the number of qubits n needed
n = len("{0:b}".format(a))
n2 = len("{0:b}".format(b))
if n2 > n:
    n = n2

# classical register with n+1 bits.
c = q_program.create_classical_register("cr", n+1)
# quantum registers
qa = q_program.create_quantum_register("qa", n) # a qubits
qb = q_program.create_quantum_register("qb", n+1) # b qubits
# if n = 1, no need of carry register
if n == 1:
    qcar = 0
    # quantum circuit involving the quantum registers and the classical register
    addnbit_circuit = q_program.create_circuit("add", [qa, qb],[c])
else:
    qcar = q_program.create_quantum_register("qcar", n-1) # carry qubits
    # quantum circuit involving the quantum registers and the classical register
    addnbit_circuit = q_program.create_circuit("add", [qa, qb, qcar],[c])

# create the state containing a
number_state(addnbit_circuit, qa, a, n)
# create the state containing b
number_state(addnbit_circuit, qb, b, n)

# addition
addition_nbit(addnbit_circuit, qa, qb, qcar, n)

# measurements to see the result
for i in range(n+1):
    addnbit_circuit.measure(qb[i], c[i])

# compile and execute the quantum program in the backend
result = q_program.execute(["add"], backend=backend, shots=1024, timeout=999999)

# show the results.
print(result)
print(result.get_data("add"))
counts = result.get_counts("add")
plot_histogram(counts)
print("Backend:", backend)
print("Highest probability outcome: {}".format(int(max(counts, key = lambda x: counts[x]).replace(" ", ""), 2)))
```

We indeed see that the only appearing outcome is $23\,(10111)$ when we execute the code on local_qasm_simulator.

## Modular addition
As you may already expect, as we increase the complexity of the arithmetic operations we also increase the complexity of the circuit network. Our next goal is to build a network that effects

\begin{align}
| a, b \rangle\, \rightarrow \, | a, a+b\, mod \,N \rangle
\end{align}

where $a+b < 2N$ (this condition is enough in order to build the following arithmetic operations). The approach consists essentially on taking the output of the plain adder network and then substracting $N$, depending on whether the value $a+b$ is bigger or smaller than $N$. In order to do so, we use a temporary qubit which indicates whether there has been an overflow in the substraction or not, and adding back $N$ depending on this. The last part of the network takes care of reseting every register (except the second register which contains the result) to its initial state.

The total number of qubits needed is $5n + 2$.

### Implementation in the quantum simulator. Example 3.
The third example is the implementation of the modular addition of two $n$-qubit numbers $a$ and $b$ (modulo $N$). This program can be executed in the quantum simulator.

```python
# checking the version of PYTHON; only support > 3.5
import sys
if sys.version_info < (3,5):
    raise Exception('Please use Python version 3.5 or greater.')

# importing QISKit
from qiskit import QuantumProgram

# import basic plotting tools
from qiskit.tools.visualization import plot_histogram

# create a QuantumProgram object instance.
q_program = QuantumProgram()

# backend
backend = 'local_qasm_simulator'
```

Now we define again the different modules that we are going to use repeatedly, as well as the modular addition algorithm for two $n$-qubit numbers. This time we also have to substract or use conditional adders (in order to substract $N$ or not), therefore we have to implement new modules such as the conditional plain addition or the plain substraction.

```python
def carry(circuit, q0, q1, q2, q3):
    "carry module"
    circuit.ccx(q1, q2, q3)
    circuit.cx(q1, q2)
    circuit.ccx(q0, q2, q3)

def carry_inv(circuit, q0, q1, q2, q3):
    "carry module running backwards"
    circuit.ccx(q0, q2, q3)
    circuit.cx(q1, q2)
    circuit.ccx(q1, q2, q3)

def summation(circuit, q0, q1, q2):
    "summation module"
    circuit.cx(q1, q2)
    circuit.cx(q0, q2)

def summation_inv(circuit, q0, q1, q2):
    "summation module running backwards"
    circuit.cx(q0, q2)
    circuit.cx(q1, q2)

# quantum plain addition algorithm for n-qubit numbers
def addition_nbit(circuit, qa, qb, qcar, n):
    if n == 1:
        circuit.ccx(qa[0], qb[0], qb[1])
        circuit.cx(qa[0], qb[0])
    else:
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])
        for i in range(n-2):
            carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1])
        carry(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n])
        circuit.cx(qa[n-1], qb[n-1])
        for i in range(n-1, 1, -1):
            summation(circuit, qcar[i-1], qa[i], qb[i])
            carry_inv(circuit, qcar[i-2], qa[i-1], qb[i-1], qcar[i-1])
        summation(circuit, qcar[0], qa[1], qb[1])
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])

# quantum plain substraction algorithm for n-qubit numbers
def subs_nbit(circuit, qa, qb, qcar, n):
    "same circuit as the plain addition but going backwards"
    if n == 1:
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qb[1])
    else:
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])
        summation_inv(circuit, qcar[0], qa[1], qb[1])
        for i in range(n-2):
            carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1])
            summation_inv(circuit, qcar[i+1], qa[i+2], qb[i+2])
        circuit.cx(qa[n-1], qb[n-1])
        carry_inv(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n])
        for i in range(n-2, 0, -1):
            carry_inv(circuit, qcar[i-1], qa[i], qb[i], qcar[i])
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])

def cond_toffoli(circuit, qcond, q1, q2, q3):
    "toffoli gate conditioned by an external qubit"
    circuit.h(q3)
    circuit.ccx(qcond, q2, q3)
    circuit.tdg(q3)
    circuit.ccx(qcond, q1, q3)
    circuit.t(q3)
    circuit.ccx(qcond, q2, q3)
    circuit.tdg(q3)
    circuit.ccx(qcond, q1, q3)
    circuit.t(q3)
    circuit.h(q3)
    circuit.t(q2)
    circuit.ccx(qcond, q1, q2)
    circuit.t(q1)
    circuit.tdg(q2)
    circuit.ccx(qcond, q1, q2)

def cond_carry(circuit, q0, q1, q2, q3, qcond):
    "conditional carry module"
    cond_toffoli(circuit, qcond, q1, q2, q3)
    circuit.ccx(qcond, q1, q2)
    cond_toffoli(circuit, qcond, q0, q2, q3)

def cond_carry_inv(circuit, q0, q1, q2, q3, qcond):
    "conditional carry module running backwards"
    cond_toffoli(circuit, qcond, q0, q2, q3)
    circuit.ccx(qcond, q1, q2)
    cond_toffoli(circuit, qcond, q1, q2, q3)

def cond_summation(circuit, q0, q1, q2, qcond):
    "conditional summation module"
    circuit.ccx(qcond, q1, q2)
    circuit.ccx(qcond, q0, q2)

def cond_summation_inv(circuit, q0, q1, q2, qcond):
    "conditional summation module running backwards"
    circuit.ccx(qcond, q0, q2)
    circuit.ccx(qcond, q1, q2)

# quantum conditional plain addition algorithm for n-qubit numbers
def cond_addition_nbit(circuit, qa, qb, qcar, qcond, n):
    "plain addition algorithm conditioned by an external qubit"
    if n == 1:
        cond_toffoli(circuit, qcond[0], qa[0], qb[0], qb[1])
        circuit.ccx(qcond[0], qa[0], qb[0])
    else:
        cond_toffoli(circuit, qcond[0], qa[0], qb[0], qcar[0])
        circuit.ccx(qcond[0], qa[0], qb[0])
        for i in range(n-2):
            cond_carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1], qcond[0])
        cond_carry(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n], qcond[0])
        circuit.ccx(qcond[0], qa[n-1], qb[n-1])
        for i in range(n-1, 1, -1):
            cond_summation(circuit, qcar[i-1], qa[i], qb[i], qcond[0])
            cond_carry_inv(circuit, qcar[i-2], qa[i-1], qb[i-1], qcar[i-1], qcond[0])
        cond_summation(circuit, qcar[0], qa[1], qb[1], qcond[0])
        circuit.ccx(qcond[0], qa[0], qb[0])
        cond_toffoli(circuit, qcond[0], qa[0], qb[0], qcar[0])
        circuit.ccx(qcond[0], qa[0], qb[0])

# quantum conditional plain substraction algorithm for n-qubit numbers
def cond_subs_nbit(circuit, qa, qb, qcar, qcond, n):
    "same circuit as the conditional plain addition but going backwards"
    if n == 1:
        circuit.ccx(qcond[0], qa[0], qb[0])
        cond_toffoli(circuit, qcond[0], qa[0], qb[0], qb[1])
    else:
        circuit.ccx(qcond[0], qa[0], qb[0])
        cond_toffoli(circuit, qcond[0], qa[0], qb[0], qcar[0])
        circuit.ccx(qcond[0], qa[0], qb[0])
        cond_summation_inv(circuit, qcar[0], qa[1], qb[1], qcond[0])
        for i in range(n-2):
            cond_carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1], qcond[0])
            cond_summation_inv(circuit, qcar[i+1], qa[i+2], qb[i+2], qcond[0])
        circuit.ccx(qcond[0], qa[n-1], qb[n-1])
        cond_carry_inv(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n], qcond[0])
        for i in range(n-2, 0, -1):
            cond_carry_inv(circuit, qcar[i-1], qa[i], qb[i], qcar[i], qcond[0])
        circuit.ccx(qcond[0], qa[0], qb[0])
        cond_toffoli(circuit, qcond[0], qa[0], qb[0], qcar[0])

# quantum modular addition algorithm for n-qubit numbers
def mod_addition_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, n):
    addition_nbit(circuit, qa, qb, qcar, n)
    subs_nbit(circuit, qN, qb, qcar, n)
    circuit.x(qb[n])
    circuit.cx(qb[n], qtemp[0])
    circuit.x(qb[n])
    cond_subs_nbit(circuit, qNtemp, qN, qcar, qtemp, n)
    addition_nbit(circuit, qN, qb, qcar, n)
    cond_addition_nbit(circuit, qNtemp, qN, qcar, qtemp, n)
    subs_nbit(circuit, qa, qb, qcar, n)
    circuit.cx(qb[n], qtemp[0])
    addition_nbit(circuit, qa, qb, qcar, n)
```

Now we define the function that creates the states containing $a$, $b$ and $N$.

```python
# n-qubit number input state
def number_state(circuit, q, x, n):
    # integer to binary
    x = "{0:b}".format(x)
    x = x.zfill(n)
    # creating the state
    for i in range(n):
        if int(x[n-1-i]) == 1:
            circuit.x(q[i])
```

Let's implement the modular addition of two numbers $a+b \, mod \, N$, for instance $2+3 \,mod\, 3$, that should return $2 \, (010)$:

```python
# we define the values
a = 2
b = 3
N = 3

# computing number of qubits n needed
n = len("{0:b}".format(a))
n2 = len("{0:b}".format(b))
n3 = len("{0:b}".format(N))
if n2 > n:
    n = n2
if n3 > n:
    n = n3

# classical register with n+1 bits.
c = q_program.create_classical_register("cr", n+1)
# quantum registers
qa = q_program.create_quantum_register("qa", n) # a qubits
qb = q_program.create_quantum_register("qb", n+1) # b qubits
qN = q_program.create_quantum_register("qN", n+1) # N qubits
qNtemp = q_program.create_quantum_register("qNtemp", n) # temporary N qubits
qtemp = q_program.create_quantum_register("qtemp", 1) # temporary qubit
# if n = 1, no need of carry register
if n == 1:
    qcar = 0
    # quantum circuit involving the quantum registers and the classical register
    mod_add_circuit = q_program.create_circuit("mod_add", [qa, qb, qN, qNtemp, qtemp],[c])
else:
    qcar = q_program.create_quantum_register("qcar", n-1) # carry qubits
    # quantum circuit involving the quantum registers and the classical register
    mod_add_circuit = q_program.create_circuit("mod_add", [qa, qb, qN, qcar, qNtemp, qtemp],[c])

# create the state containing 'a'
number_state(mod_add_circuit, qa, a, n)
# create the state containing 'b'
number_state(mod_add_circuit, qb, b, n)
# create the state containing 'N'
number_state(mod_add_circuit, qN, N, n)
# create the temporary state containing 'N'
number_state(mod_add_circuit, qNtemp, N, n)

# modular addition
mod_addition_nbit(mod_add_circuit, qa, qb, qN, qNtemp, qcar, qtemp, n)

# measurements to see the result
for i in range(n+1):
    mod_add_circuit.measure(qb[i], c[i])

# compile and execute the quantum program in the backend
result = q_program.execute(["mod_add"], backend=backend, shots=1024, max_credits=3, wait=10, timeout=999999)

# show the results.
print(result)
print(result.get_data("mod_add"))
counts = result.get_counts("mod_add")
plot_histogram(counts)
print("Backend:", backend)
print("Highest probability outcome: {}".format(int(max(counts, key = lambda x: counts[x]).replace(" ", ""), 2)))
```

Indeed the result is what we were expecting, $2\, (010)$.

## Controlled modular multiplication
The modular multiplication can be implemented by repeated conditional additions modulo $N$: $ax = 2^0 ax_0 + 2^1 ax_1 + ... + 2^{n-1} ax_{n-1}$. We start from a register initiated in the state $|0\rangle$. Then, the network consists in adding conditionally the value $2^i a$, depending on the state of the qubit $|x_i\rangle$. Anyway, in order to implement the modular exponentiation, we need a more slightly complicated circuit, by the fact that we want the multiplication be effected conditionally depeding on the value of some external qubit $|c\rangle$. If $|c\rangle = |1\rangle$

\begin{align}
|c; x,0\rangle \, \rightarrow \,  |c;x, a \times x\; mod\, N\rangle
\end{align}

This is done by applying Toffoli gates to $|x_i\rangle$ and control qubits $|c\rangle$ and the corresponding target qubit. The resetting of the register to its initial state is done by appliying the same Toffoli gates again. If the control qubit is $|0\rangle$, no operations are done, giving the state $|c; x,0\rangle$. Since we want the state to be $|c;x,x\rangle$ we copy the values of the input register to the result register. This is done in the final part of the network.

However, this circuit requires that the value to be multiplied $a$ be hardwired into the circuit.

\begin{align}
a \times x = (a_{n-1} 2^{n-1} + ... + a_0 2^0)\times (x_{n-1} 2^{n-1} + ... + x_0 2^0) = 2^{2n-2}(a_{n-1} x_{n-1}) + ...+ 2^{n-1}(a_0 x_{n-1} + a_1 x_{n-2} + .... + a_{n-2} x_1 + a_{n-1} x_0) + ... + 2^0 (a_0 x_0).
\end{align}

Notice that the indices of each pair of qubits $a_i x_j$ indicate the respective power of $2$ ($2^{i+j}$). It is important to notice that $2^{i+j}$ can exceed our modulus N. Thus, we classicaly compute this value at each step and create a temporary register to hold these values.

Summarizing, the controlled multiplication proceed as follows:
1. We begin with $n$-qubits inputs $a$ and $x$ and examine each qubit individually. We apply a three-Toffoli gate to the qubits $a_i$ and $x_j$. If both qubits along with the control qubit are $|1\rangle$ we XOR the incoming value $2^{i+j} \, mod \, N$ to the temporary register.
2. The result is then fed into an addition modulo N module which adds the result to the register holding a running total. After this, the temporary register is set to $|0\rangle$.

The total number of qubits needed is $7n + 3$.

### Implementation in the quantum simulator. Example 4.

The fourth example is the implementation of the controlled modular multiplication of two $n$-qubit numbers $a$ and $x$. This program can be executed in the quantum simulator.

```python
# checking the version of PYTHON; only support > 3.5
import sys
if sys.version_info < (3,5):
    raise Exception('Please use Python version 3.5 or greater.')

# importing QISKit
from qiskit import QuantumProgram

# import basic plotting tools
from qiskit.tools.visualization import plot_histogram

# create a QuantumProgram object instance.
q_program = QuantumProgram()

# backend
backend = 'local_qasm_simulator'
```

We define the same modules as in the modular addition, as well as the controlled modular multiplication algorithm for $n$-qubit numbers.

```python
def carry(circuit, q0, q1, q2, q3):
    "carry module"
    circuit.ccx(q1, q2, q3)
    circuit.cx(q1, q2)
    circuit.ccx(q0, q2, q3)

def carry_inv(circuit, q0, q1, q2, q3):
    "carry module running backwards"
    circuit.ccx(q0, q2, q3)
    circuit.cx(q1, q2)
    circuit.ccx(q1, q2, q3)

def summation(circuit, q0, q1, q2):
    "summation module"
    circuit.cx(q1, q2)
    circuit.cx(q0, q2)

def summation_inv(circuit, q0, q1, q2):
    "summation module running backwards"
    circuit.cx(q0, q2)
    circuit.cx(q1, q2)

# quantum plain addition algorithm for n-qubit numbers
def addition_nbit(circuit, qa, qb, qcar, n):
    if n == 1:
        circuit.ccx(qa[0], qb[0], qb[1])
        circuit.cx(qa[0], qb[0])
    else:
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])
        for i in range(n-2):
            carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1])
        carry(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n])
        circuit.cx(qa[n-1], qb[n-1])
        for i in range(n-1, 1, -1):
            summation(circuit, qcar[i-1], qa[i], qb[i])
            carry_inv(circuit, qcar[i-2], qa[i-1], qb[i-1], qcar[i-1])
        summation(circuit, qcar[0], qa[1], qb[1])
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])

# quantum plain substraction algorithm for n-qubit numbers
def subs_nbit(circuit, qa, qb, qcar, n):
    "same circuit as the addition but going backwards"
    if n == 1:
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qb[1])
    else:
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])
        summation_inv(circuit, qcar[0], qa[1], qb[1])
        for i in range(n-2):
            carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1])
            summation_inv(circuit, qcar[i+1], qa[i+2], qb[i+2])
        circuit.cx(qa[n-1], qb[n-1])
        carry_inv(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n])
        for i in range(n-2, 0, -1):
            carry_inv(circuit, qcar[i-1], qa[i], qb[i], qcar[i])
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])

def cond_toffoli(circuit, qcond, q1, q2, q3):
    "toffoli gate conditioned by an external qubit"
    circuit.h(q3)
    circuit.ccx(qcond, q2, q3)
    circuit.tdg(q3)
    circuit.ccx(qcond, q1, q3)
    circuit.t(q3)
    circuit.ccx(qcond, q2, q3)
    circuit.tdg(q3)
    circuit.ccx(qcond, q1, q3)
    circuit.t(q3)
    circuit.h(q3)
    circuit.t(q2)
    circuit.ccx(qcond, q1, q2)
    circuit.t(q1)
    circuit.tdg(q2)
    circuit.ccx(qcond, q1, q2)

def cond_carry(circuit, q0, q1, q2, q3, qcond):
    "conditional carry module"
    cond_toffoli(circuit, qcond, q1, q2, q3)
    circuit.ccx(qcond, q1, q2)
    cond_toffoli(circuit, qcond, q0, q2, q3)

def cond_carry_inv(circuit, q0, q1, q2, q3, qcond):
    "conditional carry module running backwards"
    cond_toffoli(circuit, qcond, q0, q2, q3)
    circuit.ccx(qcond, q1, q2)
    cond_toffoli(circuit, qcond, q1, q2, q3)

def cond_summation(circuit, q0, q1, q2, qcond):
    "conditional summation module"
    circuit.ccx(qcond, q1, q2)
    circuit.ccx(qcond, q0, q2)

def cond_summation_inv(circuit, q0, q1, q2, qcond):
    "conditional summation module running backwards"
    circuit.ccx(qcond, q0, q2)
    circuit.ccx(qcond, q1, q2)

# quantum conditional plain addition algorithm for n-qubit numbers
def cond_addition_nbit(circuit, qa, qb, qcar, qcond, n):
    if n == 1:
        cond_toffoli(circuit, qcond, qa[0], qb[0], qb[1])
        circuit.ccx(qcond, qa[0], qb[0])
    else:
        cond_toffoli(circuit, qcond, qa[0], qb[0], qcar[0])
        circuit.ccx(qcond, qa[0], qb[0])
        for i in range(n-2):
            cond_carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1], qcond)
        cond_carry(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n], qcond)
        circuit.ccx(qcond, qa[n-1], qb[n-1])
        for i in range(n-1, 1, -1):
            cond_summation(circuit, qcar[i-1], qa[i], qb[i], qcond)
            cond_carry_inv(circuit, qcar[i-2], qa[i-1], qb[i-1], qcar[i-1], qcond)
        cond_summation(circuit, qcar[0], qa[1], qb[1], qcond)
        circuit.ccx(qcond, qa[0], qb[0])
        cond_toffoli(circuit, qcond, qa[0], qb[0], qcar[0])
        circuit.ccx(qcond, qa[0], qb[0])

# quantum conditional plain substraction algorithm for n-qubit numbers
def cond_subs_nbit(circuit, qa, qb, qcar, qcond, n):
    "same circuit as the conditional plain addition but going backwards"
    if n == 1:
        circuit.ccx(qcond, qa[0], qb[0])
        cond_toffoli(circuit, qcond, qa[0], qb[0], qb[1])
    else:
        circuit.ccx(qcond, qa[0], qb[0])
        cond_toffoli(circuit, qcond, qa[0], qb[0], qcar[0])
        circuit.ccx(qcond, qa[0], qb[0])
        cond_summation_inv(circuit, qcar[0], qa[1], qb[1], qcond)
        for i in range(n-2):
            cond_carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1], qcond)
            cond_summation_inv(circuit, qcar[i+1], qa[i+2], qb[i+2], qcond)
        circuit.ccx(qcond, qa[n-1], qb[n-1])
        cond_carry_inv(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n], qcond)
        for i in range(n-2, 0, -1):
            cond_carry_inv(circuit, qcar[i-1], qa[i], qb[i], qcar[i], qcond)
        circuit.ccx(qcond, qa[0], qb[0])
        cond_toffoli(circuit, qcond, qa[0], qb[0], qcar[0])

# quantum modular addition algorithm for n-qubit numbers
def mod_addition_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, n):
    addition_nbit(circuit, qa, qb, qcar, n)
    subs_nbit(circuit, qN, qb, qcar, n)
    circuit.x(qb[n])
    circuit.cx(qb[n], qtemp)
    circuit.x(qb[n])
    cond_subs_nbit(circuit, qNtemp, qN, qcar, qtemp, n)
    addition_nbit(circuit, qN, qb, qcar, n)
    cond_addition_nbit(circuit, qNtemp, qN, qcar, qtemp, n)
    subs_nbit(circuit, qa, qb, qcar, n)
    circuit.cx(qb[n], qtemp)
    addition_nbit(circuit, qa, qb, qcar, n)

# quantum controlled modular multiplication algorithm for n-qubit numbers
def cont_mod_mult_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, qtempst, qX, qext, N, n):
    for i in range(n):
        for j in range(n):
            classical_mod = (2**(i+j))%N
            cond_number_state(circuit, qtempst, classical_mod, qext, qa[i], qX[j], n)
            mod_addition_nbit(circuit, qtempst, qb, qN, qNtemp, qcar, qtemp, n)
            cond_number_state(circuit, qtempst, classical_mod, qext, qa[i], qX[j], n)
    circuit.x(qext)
    cond_addition_nbit(circuit, qX, qb, qcar, qext, n)
    circuit.x(qext)
```

We define the function that creates the different states. This time we also define another function, what does essentially the same, but controlled by 3 control qubits.

```python
# n-qubit number input state
def number_state(circuit, q, x, n):
    # integer to binary
    x = "{0:b}".format(x)
    x = x.zfill(n)
    # creating the state
    for i in range(n):
        if int(x[n-1-i]) == 1:
            circuit.x(q[i])

# n-qubit number input state, controlled by 2 control qubits
def cond_number_state(circuit, q, x, ext, control1, control2, n):
    # integer to binary
    x = "{0:b}".format(x)
    x = x.zfill(n)
    # creating the state
    for i in range(n):
        if int(x[n-1-i]) == 1:
            cond_toffoli(circuit, ext, control1, control2, q[i])
```

Let's implement now the controlled modular multiplication of two numbers $a \times x \, mod \, N$, for instance $1 \times 1 \,mod\, 1$ (in order to avoid long computing times, but you can try with two-qubit numbers), that should return $0 \, (00$), if the control qubit is $|1\rangle$ (otherwise the result would be $1 \, (01)$):

```python
# we define the values
a = 1
x = 1
N = 1

# computing number of qubits n needed
n = len("{0:b}".format(a))
n2 = len("{0:b}".format(x))
n3 = len("{0:b}".format(N))
if n2 > n:
    n = n2
if n3 > n:
    n = n3

# classical register with n+1 bits.
c = q_program.create_classical_register("cr", n+1)
# quantum registers
qa = q_program.create_quantum_register("qa", n) # a qubits
qb = q_program.create_quantum_register("qb", n+1) # result register
qN = q_program.create_quantum_register("qN", n+1) # N qubits
qNtemp = q_program.create_quantum_register("qNtemp", n) # temporary N qubits
qtemp = q_program.create_quantum_register("qtemp", 1) # temporary qubit
qtempst = q_program.create_quantum_register("qtempst", n) # temporary register
qX = q_program.create_quantum_register("qX", n) # x register
qext = q_program.create_quantum_register("qext", 1)
# if n = 1, no need of carry register
if n == 1:
    qcar = 0
    # quantum circuit involving the quantum registers and the classical register
    mod_mult_circuit = q_program.create_circuit("mod_mult", [qa, qb, qN, qNtemp, qtemp, qtempst, qX, qext],[c])
else:
    qcar = q_program.create_quantum_register("qcar", n-1) # carry qubits
    # quantum circuit involving the quantum register and the classical register
    mod_mult_circuit = q_program.create_circuit("mod_mult", [qa, qb, qN, qcar, qNtemp, qtemp, qtempst, qX, qext],[c])

# create the state containing 'a'
number_state(mod_mult_circuit, qa, a, n)
# create the state containing 'b'
number_state(mod_mult_circuit, qX, x, n)
# create the state containing 'N'
number_state(mod_mult_circuit, qN, N, n+1)
# create a temporary state containing 'N'
number_state(mod_mult_circuit, qNtemp, N, n)

mod_mult_circuit.x(qext[0]) # we set the control qubit to |1>

# controlled modular multiplication
cont_mod_mult_nbit(mod_mult_circuit, qa, qb, qN, qNtemp, qcar, qtemp[0], qtempst, qX, qext[0], N, n)

# measurements to see the result
for i in range(n+1):
    mod_mult_circuit.measure(qb[i], c[i])

# compile and execute the quantum program in the backend
result = q_program.execute(["mod_mult"], backend=backend, shots=1024, max_credits=3, wait=10, timeout=999999)

# show the results.
print(result)
print(result.get_data("mod_mult"))
counts = result.get_counts("mod_mult")
plot_histogram(counts)
print("Backend:", backend)
print("Highest probability outcome: {}".format(int(max(counts, key = lambda x: counts[x]).replace(" ", ""), 2)))
```

We indeed obtain the expected result $0 \, (00)$.

## Modular exponentiation
Using the previous constructions, we can finally implement the exponentiation modulo $N$: $a^x \,mod \,N$. The value $a^x$ can be written as $a^x = a^{2^{0}x_0} · a^{2^{1}x_1} · ... · a^{2^{m-1}x_{m-1}}$. The modular exponentiation can be implemented by initially setting a register to $|1\rangle$ and then effecting $n$ modular multiplications by $a^{2^i}$ depending on the value of the qubit $|x_i\rangle$. Let's look into the operations carefully. If $x_i = 1$:

\begin{align}
|a^{2^{0}x_{0} + ... + 2^{i-1}x_{i-1}}, 0 \rangle \;\rightarrow\; |a^{2^{0}x_{0} + ... + 2^{i-1}x_{i-1}}, a^{2^{0}x_{0} + ... + 2^{i-1}x_{i-1}} · a^{2^i}\rangle
\end{align}

otherwise, if $x_i = 0$:

\begin{align}
|a^{2^{0}x_{0} + ... + 2^{i-1}x_{i-1}}, 0 \rangle \;\rightarrow\; |a^{2^{0}x_{0} + ... + 2^{i-1}x_{i-1}}, a^{2^{0}x_{0} + ... + 2^{i-1}x_{i-1}}\rangle \,.
\end{align}

In both cases the result can be written as $|a^{2^{0}x_{0} + ... + 2^{i-1}x_{i-1}}, a^{2^{0}x_{0} + ... + 2^{i}x_{i}}\rangle$. We must take care about the accumulation of intermediate data. To avoid this partial information generated we run backwards a controlled multiplication network with the value $a^{-2^i} \,mod\, N$. This quantity is precomputed classically, but $a$ and $N$ must be coprimes.

The total number of qubits needed is $7n + 3 + n_x$, where $n_x$ are the number of qubits of the value $x$.

### Implementation in the quantum simulator. Example 5.

As you may expect, we do not want to finish this QISKIT tutorial of basic arithmetic operations without the keystone of the Shor's factorization algorithm. The final example is the implementation of the modular exponentiation $a^x \; mod\, N$. This program can be executed in the quantum simulator.

```python
# checking the version of PYTHON; only support > 3.5
import sys
if sys.version_info < (3,5):
    raise Exception('Please use Python version 3.5 or greater.')

# importing QISKit
from qiskit import QuantumProgram

# import basic plotting tools
from qiskit.tools.visualization import plot_histogram

# create a QuantumProgram object instance.
q_program = QuantumProgram()

# backend
backend = 'local_qasm_simulator'
```

This time we have to run backwards the controlled modular multiplication block, therefore we will need to define new modules such as running backwards the modular addition.

```python
def carry(circuit, q0, q1, q2, q3):
    "carry module"
    circuit.ccx(q1, q2, q3)
    circuit.cx(q1, q2)
    circuit.ccx(q0, q2, q3)

def carry_inv(circuit, q0, q1, q2, q3):
    "carry module running backwards"
    circuit.ccx(q0, q2, q3)
    circuit.cx(q1, q2)
    circuit.ccx(q1, q2, q3)

def summation(circuit, q0, q1, q2):
    "summation module"
    circuit.cx(q1, q2)
    circuit.cx(q0, q2)

def summation_inv(circuit, q0, q1, q2):
    "summation module running backwards"
    circuit.cx(q0, q2)
    circuit.cx(q1, q2)

# quantum plain addition algorithm for n-qubit numbers
def addition_nbit(circuit, qa, qb, qcar, n):
    if n == 1:
        circuit.ccx(qa[0], qb[0], qb[1])
        circuit.cx(qa[0], qb[0])
    else:
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])
        for i in range(n-2):
            carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1])
        carry(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n])
        circuit.cx(qa[n-1], qb[n-1])
        for i in range(n-1, 1, -1):
            summation(circuit, qcar[i-1], qa[i], qb[i])
            carry_inv(circuit, qcar[i-2], qa[i-1], qb[i-1], qcar[i-1])
        summation(circuit, qcar[0], qa[1], qb[1])
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])

# quantum plain substraction algorithm for n-qubit numbers
def subs_nbit(circuit, qa, qb, qcar, n):
    "same as the plain addition but running backwards"
    if n == 1:
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qb[1])
    else:
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])
        circuit.cx(qa[0], qb[0])
        summation_inv(circuit, qcar[0], qa[1], qb[1])
        for i in range(n-2):
            carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1])
            summation_inv(circuit, qcar[i+1], qa[i+2], qb[i+2])
        circuit.cx(qa[n-1], qb[n-1])
        carry_inv(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n])
        for i in range(n-2, 0, -1):
            carry_inv(circuit, qcar[i-1], qa[i], qb[i], qcar[i])
        circuit.cx(qa[0], qb[0])
        circuit.ccx(qa[0], qb[0], qcar[0])

def cond_toffoli(circuit, qcond, q1, q2, q3):
    "conditional toffoli gate"
    circuit.h(q3)
    circuit.ccx(qcond, q2, q3)
    circuit.tdg(q3)
    circuit.ccx(qcond, q1, q3)
    circuit.t(q3)
    circuit.ccx(qcond, q2, q3)
    circuit.tdg(q3)
    circuit.ccx(qcond, q1, q3)
    circuit.t(q3)
    circuit.h(q3)
    circuit.t(q2)
    circuit.ccx(qcond, q1, q2)
    circuit.t(q1)
    circuit.tdg(q2)
    circuit.ccx(qcond, q1, q2)

def cond_carry(circuit, q0, q1, q2, q3, qcond):
    "conditional carry module"
    cond_toffoli(circuit, qcond, q1, q2, q3)
    circuit.ccx(qcond, q1, q2)
    cond_toffoli(circuit, qcond, q0, q2, q3)

def cond_carry_inv(circuit, q0, q1, q2, q3, qcond):
    "conditional carry module running backwards"
    cond_toffoli(circuit, qcond, q0, q2, q3)
    circuit.ccx(qcond, q1, q2)
    cond_toffoli(circuit, qcond, q1, q2, q3)

def cond_summation(circuit, q0, q1, q2, qcond):
    "conditional summation"
    circuit.ccx(qcond, q1, q2)
    circuit.ccx(qcond, q0, q2)

def cond_summation_inv(circuit, q0, q1, q2, qcond):
    "conditional summation running backwards"
    circuit.ccx(qcond, q0, q2)
    circuit.ccx(qcond, q1, q2)

# quantum conditional plain addition algorithm for n-qubit numbers
def cond_addition_nbit(circuit, qa, qb, qcar, qcond, n):
    if n == 1:
        cond_toffoli(circuit, qcond, qa[0], qb[0], qb[1])
        circuit.ccx(qcond, qa[0], qb[0])
    else:
        cond_toffoli(circuit, qcond, qa[0], qb[0], qcar[0])
        circuit.ccx(qcond, qa[0], qb[0])
        for i in range(n-2):
            cond_carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1], qcond)
        cond_carry(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n], qcond)
        circuit.ccx(qcond, qa[n-1], qb[n-1])
        for i in range(n-1, 1, -1):
            cond_summation(circuit, qcar[i-1], qa[i], qb[i], qcond)
            cond_carry_inv(circuit, qcar[i-2], qa[i-1], qb[i-1], qcar[i-1], qcond)
        cond_summation(circuit, qcar[0], qa[1], qb[1], qcond)
        circuit.ccx(qcond, qa[0], qb[0])
        cond_toffoli(circuit, qcond, qa[0], qb[0], qcar[0])
        circuit.ccx(qcond, qa[0], qb[0])

# quantum conditional plain substraction algorithm for n-qubit numbers
def cond_subs_nbit(circuit, qa, qb, qcar, qcond, n):
    "same as conditional plain addition but running backwards"
    if n == 1:
        circuit.ccx(qcond, qa[0], qb[0])
        cond_toffoli(circuit, qcond, qa[0], qb[0], qb[1])
    else:
        circuit.ccx(qcond, qa[0], qb[0])
        cond_toffoli(circuit, qcond, qa[0], qb[0], qcar[0])
        circuit.ccx(qcond, qa[0], qb[0])
        cond_summation_inv(circuit, qcar[0], qa[1], qb[1], qcond)
        for i in range(n-2):
            cond_carry(circuit, qcar[i], qa[i+1], qb[i+1], qcar[i+1], qcond)
            cond_summation_inv(circuit, qcar[i+1], qa[i+2], qb[i+2], qcond)
        circuit.ccx(qcond, qa[n-1], qb[n-1])
        cond_carry_inv(circuit, qcar[n-2], qa[n-1], qb[n-1], qb[n], qcond)
        for i in range(n-2, 0, -1):
            cond_carry_inv(circuit, qcar[i-1], qa[i], qb[i], qcar[i], qcond)
        circuit.ccx(qcond, qa[0], qb[0])
        cond_toffoli(circuit, qcond, qa[0], qb[0], qcar[0])

# quantum modular addition algorithm for n-qubit numbers
def mod_addition_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, n):
    addition_nbit(circuit, qa, qb, qcar, n)
    subs_nbit(circuit, qN, qb, qcar, n)
    circuit.x(qb[n])
    circuit.cx(qb[n], qtemp)
    circuit.x(qb[n])
    cond_subs_nbit(circuit, qNtemp, qN, qcar, qtemp, n)
    addition_nbit(circuit, qN, qb, qcar, n)
    cond_addition_nbit(circuit, qNtemp, qN, qcar, qtemp, n)
    subs_nbit(circuit, qa, qb, qcar, n)
    circuit.cx(qb[n], qtemp)
    addition_nbit(circuit, qa, qb, qcar, n)

# quantum modular substraction algorithm for n-qubit numbers
def mod_subs_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, n):
    "same as modular addition but running backwards"
    subs_nbit(circuit, qa, qb, qcar, n)
    circuit.cx(qb[n], qtemp)
    addition_nbit(circuit, qa, qb, qcar, n)
    cond_subs_nbit(circuit, qNtemp, qN, qcar, qtemp, n)
    subs_nbit(circuit, qN, qb, qcar, n)
    cond_addition_nbit(circuit, qNtemp, qN, qcar, qtemp, n)
    circuit.x(qb[n])
    circuit.cx(qb[n], qtemp)
    circuit.x(qb[n])
    addition_nbit(circuit, qN, qb, qcar, n)
    subs_nbit(circuit, qa, qb, qcar, n)

# quantum controlled modular multiplication algorithm for n-qubit numbers
def cont_mod_mult_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, qtempst, qX, qext, N, n):
    for i in range(n):
        for j in range(n):
            classical_mod = (2**(i+j))%N
            cond_number_state(circuit, qtempst, classical_mod, qext, qa[i], qX[j], n)
            mod_addition_nbit(circuit, qtempst, qb, qN, qNtemp, qcar, qtemp, n)
            cond_number_state(circuit, qtempst, classical_mod, qext, qa[i], qX[j], n)
    circuit.x(qext)
    cond_addition_nbit(circuit, qX, qb, qcar, qext, n)
    circuit.x(qext)

def cont_inv_mod_mult_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, qtempst, qX, qext, N, n):
    "same as the controlled modular multiplication but running backwards"
    circuit.x(qext)
    cond_subs_nbit(circuit, qX, qb, qcar, qext, n)
    circuit.x(qext)
    for i in range(n):
        for j in range(n):
            classical_mod = (2**(i+j))%N
            cond_number_state(circuit, qtempst, classical_mod, qext, qa[i], qX[j], n)
            mod_subs_nbit(circuit, qtempst, qb, qN, qNtemp, qcar, qtemp, n)
            cond_number_state(circuit, qtempst, classical_mod, qext, qa[i], qX[j], n)

# quantum modular exponentiation algorithm for n-qubit numbers
def mod_exp_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, qtempst, q1, qX, N, a, n):
    for k in range(len(qX)):
        clas_value = (a**(2**(k)))%N
        if k % 2 == 0:
            number_state(circuit, qa, clas_value, n)
            cont_mod_mult_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, qtempst, q1, qX[k], N, n)
            number_state(circuit, qa, clas_value, n)
            clas_value = modinv(a**(2**(k)), N)
            number_state(circuit, qa, clas_value, n)
            cont_inv_mod_mult_nbit(circuit, qa, q1, qN, qNtemp, qcar, qtemp, qtempst, qb, qX[k], N, n)
            number_state(circuit, qa, clas_value, n)
        else:
            number_state(circuit, qa, clas_value, n)
            cont_mod_mult_nbit(circuit, qa, q1, qN, qNtemp, qcar, qtemp, qtempst, qb, qX[k], N, n)
            number_state(circuit, qa, clas_value, n)
            clas_value = modinv(a**(2**(k)), N)
            number_state(circuit, qa, clas_value, n)
            cont_inv_mod_mult_nbit(circuit, qa, qb, qN, qNtemp, qcar, qtemp, qtempst, q1, qX[k], N, n)
            number_state(circuit, qa, clas_value, n)
```

We define the function that creates the different states. This time we also define another two functions, which compute the value of the modular multiplicative inverse $a^{-1} \, mod \, m$.

```python
# n-qubit number input state
def number_state(circuit, q, x, n):
    # integer to binary
    x = "{0:b}".format(x)
    x = x.zfill(n)
    # creating the state
    for i in range(n):
        if int(x[n-1-i]) == 1:
            circuit.x(q[i])

# n-qubit number input state, controlled by 2 control qubits
def cond_number_state(circuit, q, x, ext, control1, control2, n):
    # integer to binary
    x = "{0:b}".format(x)
    x = x.zfill(n)
    # creating the state
    for i in range(n):
        if int(x[n-1-i]) == 1:
            cond_toffoli(circuit, ext, control1, control2, q[i])

# efficient algorithm for computing the modular multiplicative inverse a^-1 mod m
def egcd(a, b):
    if a == 0:
        return (b, 0, 1)
    else:
        g, y, x = egcd(b % a, a)
        return (g, x - (b // a) * y, y)

def modinv(a, m):
    g, x, y = egcd(a, m)
    if g != 1:
        raise Exception('modular inverse does not exist')
    else:
        return x % m
```

Let's implement now the modular exponentiation of two $n$-qubit numbers $a$ and $N$ ($a^x \, mod \, N$), for instance $1^1 \,mod\, 1$ (in order to avoid long computing times, but you can try with two-qubit numbers), that should return $0 \, (00)$:

```python
# we define the values, a and N must be coprimes
a = 1
x = 1
N = 1

# computing number of qubits n needed
n = len("{0:b}".format(a))
n2 = len("{0:b}".format(x))
n3 = len("{0:b}".format(N))

if n3 > n:
    n = n3

# classical register with n+1 bits.
c = q_program.create_classical_register("cr", n+1)
# quantum registers
qa = q_program.create_quantum_register("qa", n) # a qubits
qb = q_program.create_quantum_register("qb", n+1) # initial state |0>
qN = q_program.create_quantum_register("qN", n+1) # N qubits
qNtemp = q_program.create_quantum_register("qNtemp", n) # temporary N qubits
qtemp = q_program.create_quantum_register("qtemp", 1) # temporary qubit
qtempst = q_program.create_quantum_register("qtempst", n) # temporary register
q1 = q_program.create_quantum_register("q1", n+1) # initial state |1>
qX = q_program.create_quantum_register("qX", n2) # x register
# if n = 1, no need of carry register
if n == 1:
    qcar = 0
    # quantum circuit involving the quantum registers and the classical register
    mod_exp_circuit = q_program.create_circuit("mod_exp", [qa, qb, qN, qNtemp, qtemp, qtempst, q1, qX],[c])
else:
    qcar = q_program.create_quantum_register("qcar", n-1) # carry qubits
    # quantum circuit involving the quantum registers and the classical register
    mod_exp_circuit = q_program.create_circuit("mod_exp", [qa, qb, qN, qcar, qNtemp, qtemp, qtempst, q1, qX],[c])

# create the initial state |1>. If N = 1, initial state is |0>
if N != 1:
    number_state(mod_exp_circuit, q1, 1, 1)
# create the state containing 'x'
number_state(mod_exp_circuit, qX, x, n2)
# create the state containing 'N'
number_state(mod_exp_circuit, qN, N, n+1)
# create a temporary state containing 'N'
number_state(mod_exp_circuit, qNtemp, N, n)

# modular exponentiation
mod_exp_nbit(mod_exp_circuit, qa, qb, qN, qNtemp, qcar, qtemp[0], qtempst, q1, qX, N, a, n)

# measurements to see the result, the result would be in one of those registers, q1 or qb
if n2 % 2 == 0:
    for i in range(n+1):
        mod_exp_circuit.measure(q1[i], c[i])
else:
    for i in range(n+1):
        mod_exp_circuit.measure(qb[i], c[i])

# compile and execute the quantum program in the backend
result = q_program.execute(["mod_exp"], backend=backend, shots=1024, max_credits=3, wait=10, timeout=9999999)

# show the results.
print(result)
print(result.get_data("mod_exp"))
counts = result.get_counts("mod_exp")
plot_histogram(counts)
print("Backend:", backend)
print("Highest probability outcome: {}".format(int(max(counts, key = lambda x: counts[x]).replace(" ", ""), 2)))
```

We indeed have obtained the expected result $0 \, (00)$.


# Getting started with TensorFlow (Eager Mode)

**Learning Objectives**
  - Understand difference between Tensorflow's two modes: Eager Execution and Graph Execution
  - Practice defining and performing basic operations on constant Tensors
  - Use Tensorflow's automatic differentiation capability

## Introduction

**Eager Execution**

Eager mode evaluates operations immediatley and return concrete values immediately. To enable eager mode simply place `tf.enable_eager_execution()` at the top of your code. We recommend using eager execution when prototyping as it is intuitive, easier to debug, and requires less boilerplate code.

**Graph Execution**

Graph mode is TensorFlow's default execution mode (although it will change to eager with TF 2.0). In graph mode operations only produce a symbolic graph which doesn't get executed until run within the context of a tf.Session(). This style of coding is less inutitive and has more boilerplate, however it can lead to performance optimizations and is particularly suited for distributing training across multiple devices. We recommend using delayed execution for performance sensitive production code.

```python
# Ensure that we have Tensorflow 1.13.1 installed.
!pip3 freeze | grep tensorflow==1.13.1 || pip3 install tensorflow==1.13.1
```

    tensorflow==1.13.1

```python
import tensorflow as tf
print(tf.__version__)
```

    1.13.1

## Eager Execution

```python
tf.enable_eager_execution()
```

### Adding Two Tensors

The value of the tensor, as well as its shape and data type are printed

```python
a = tf.constant(value = [5, 3, 8], dtype = tf.int32)
b = tf.constant(value = [3, -1, 2], dtype = tf.int32)
c = tf.add(x = a, y = b)
print(c)
```

    tf.Tensor([ 8  2 10], shape=(3,), dtype=int32)

#### Overloaded Operators
We can also perform a `tf.add()` using the `+` operator. The `/,-,*` and `**` operators are similarly overloaded with the appropriate tensorflow operation.

```python
c = a + b # this is equivalent to tf.add(a,b)
print(c)
```

    tf.Tensor([ 8  2 10], shape=(3,), dtype=int32)

### NumPy Interoperability

In addition to native TF tensors, tensorflow operations can take native python types and NumPy arrays as operands.

```python
import numpy as np

a_py = [1,2] # native python list
b_py = [3,4] # native python list

a_np = np.array(object = [1,2]) # numpy array
b_np = np.array(object = [3,4]) # numpy array

a_tf = tf.constant(value = [1,2], dtype = tf.int32) # native TF tensor
b_tf = tf.constant(value = [3,4], dtype = tf.int32) # native TF tensor

for result in [tf.add(x = a_py, y = b_py), tf.add(x = a_np, y = b_np), tf.add(x = a_tf, y = b_tf)]:
    print("Type: {}, Value: {}".format(type(result), result))
```

    Type: <class 'tensorflow.python.framework.ops.EagerTensor'>, Value: [4 6]
    Type: <class 'tensorflow.python.framework.ops.EagerTensor'>, Value: [4 6]
    Type: <class 'tensorflow.python.framework.ops.EagerTensor'>, Value: [4 6]

You can convert a native TF tensor to a NumPy array using .numpy()

```python
a_tf.numpy()
```

    array([1, 2], dtype=int32)

### Linear Regression

Now let's use low level tensorflow operations to implement linear regression.

Later in the course you'll see abstracted ways to do this using high level TensorFlow.

#### Toy Dataset

We'll model the following function:

\begin{equation}
y= 2x + 10
\end{equation}

```python
X = tf.constant(value = [1,2,3,4,5,6,7,8,9,10], dtype = tf.float32)
Y = 2 * X + 10
print("X:{}".format(X))
print("Y:{}".format(Y))
```

    X:[ 1.  2.  3.  4.  5.  6.  7.  8.  9. 10.]
    Y:[12. 14. 16. 18. 20. 22. 24. 26. 28. 30.]

#### Loss Function

Using mean squared error, our loss function is:
\begin{equation}
MSE = \frac{1}{m}\sum_{i=1}^{m}(\hat{Y}_i-Y_i)^2
\end{equation}

$\hat{Y}$ represents the vector containing our model's predictions:
\begin{equation}
\hat{Y} = w_0X + w_1
\end{equation}

```python
def loss_mse(X, Y, w0, w1):
    Y_hat = w0 * X + w1
    return tf.reduce_mean(input_tensor = (Y_hat - Y)**2)
```

#### Gradient Function

To use gradient descent we need to take the partial derivative of the loss function with respect to each of the weights. We could manually compute the derivatives, but with Tensorflow's automatic differentiation capabilities we don't have to!

During gradient descent we think of the loss as a function of the parameters $w_0$ and $w_1$. Thus, we want to compute the partial derivative with respect to these variables. The `params=[2,3]` argument tells TensorFlow to only compute derivatives with respect to the 2nd and 3rd arguments to the loss function (counting from 0, so really the 3rd and 4th).

```python
# Counting from 0, the 2nd and 3rd parameter to the loss function are our weights
grad_f = tf.contrib.eager.gradients_function(f = loss_mse, params=[2,3])
```

    WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
    For more information, please see:
      * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
      * https://github.com/tensorflow/addons
    If you depend on functionality not listed there, please file an issue.

#### Training Loop

Here we have a very simple training loop that converges. Note we are ignoring best practices like batching, creating a separate test set, and random weight initialization for the sake of simplicity.

```python
STEPS = 1000
LEARNING_RATE = .02

# Initialize weights
w0 = tf.constant(value = 0.0, dtype = tf.float32)
w1 = tf.constant(value = 0.0, dtype = tf.float32)

for step in range(STEPS):
    #1. Calculate gradients
    d_w0, d_w1 = grad_f(X, Y, w0, w1)

    #2. Update weights
    w0 = w0 - d_w0 * LEARNING_RATE
    w1 = w1 - d_w1 * LEARNING_RATE

    #3. Periodically print MSE
    if step % 100 == 0:
        print("STEP: {} MSE: {}".format(step, loss_mse(X, Y, w0, w1)))

# Print final MSE and weights
print("STEP: {} MSE: {}".format(STEPS,loss_mse(X, Y, w0, w1)))
print("w0:{}".format(round(float(w0), 4)))
print("w1:{}".format(round(float(w1), 4)))
```

    STEP: 0 MSE: 167.6111297607422
    STEP: 100 MSE: 3.5321757793426514
    STEP: 200 MSE: 0.6537718176841736
    STEP: 300 MSE: 0.12100745737552643
    STEP: 400 MSE: 0.022397063672542572
    STEP: 500 MSE: 0.004145540297031403
    STEP: 600 MSE: 0.0007674093940295279
    STEP: 700 MSE: 0.00014202017337083817
    STEP: 800 MSE: 2.628635775181465e-05
    STEP: 900 MSE: 4.86889211970265e-06
    STEP: 1000 MSE: 9.178326081382693e-07
    w0:2.0003
    w1:9.9979

## Bonus

Try modelling a non-linear function such as: $y=xe^{-x^2}$

```python
X = tf.constant(value = np.linspace(0,2,1000), dtype = tf.float32)
Y = X*np.exp(-X**2) * X

from matplotlib import pyplot as plt
%matplotlib inline
plt.plot(X, Y)
```

```python
def make_features(X):
    features = [X]
    features.append(tf.ones_like(X))  # Bias.
    features.append(tf.square(X))
    features.append(tf.sqrt(X))
    features.append(tf.exp(X))
    return tf.stack(features, axis=1)

def make_weights(n_weights):
    W = [tf.constant(value = 0.0, dtype = tf.float32) for _ in range(n_weights)]
    return tf.expand_dims(tf.stack(W),-1)

def predict(X, W):
    Y_hat = tf.matmul(X, W)
    return tf.squeeze(Y_hat, axis=-1)

def loss_mse(X, Y, W):
    Y_hat = predict(X, W)
    return tf.reduce_mean(input_tensor = (Y_hat - Y)**2)

X = tf.constant(value = np.linspace(0,2,1000), dtype = tf.float32)
Y = np.exp(-X**2) * X

grad_f = tf.contrib.eager.gradients_function(f = loss_mse, params=[2])
```

```python
STEPS = 2000
LEARNING_RATE = .01

# Weights/features.
Xf = make_features(X)
# Xf = Xf[:,0:2]  # Linear features only.
W = make_weights(Xf.get_shape()[1].value)

# For plotting
steps = []
losses = []

plt.figure()
for step in range(STEPS):
    #1. Calculate gradients
    dW = grad_f(Xf, Y, W)[0]
    #2. Update weights
    W -= dW * LEARNING_RATE
    #3. Periodically print MSE
    if step % 100 == 0:
        loss = loss_mse(Xf, Y, W)
        steps.append(step)
        losses.append(loss)
        plt.clf()
        plt.plot(steps, losses)
# Print final MSE and weights
print("STEP: {} MSE: {}".format(STEPS,loss_mse(Xf, Y, W)))

# Plot results
plt.figure()
plt.plot(X, Y, label='actual')
plt.plot(X, predict(Xf, W), label='predicted')
plt.legend()
```

Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License


```python
import sys

import numpy as np
import matplotlib.pyplot as plt

sys.path.append('../src')
import utils
```

# Machine Learning Models

Many problems in machine learing seek to build a model

$$g(a; x) \approx y$$

given a data set

$$\{(a_1, y_1), \dots, (a_m, y_m)\},$$

with components
* $a_i \in \mathbf{R}^n$ - data features,
* $y_i \in \mathbf{R}$ or $\{0, 1\}$ - data value or class,
* $g: \mathbf{R}^n \to \mathbf{R}$ or $\{0, 1\}$ - prediction function,
* $x \in \mathbf{R}^n$ - model parameters,
* $m$ - number of data points, and
* $n$ - number of data features.

We can fit a model to the given data by solving an optimization problem of the form

$$\min_x \sum_{i=1}^m f_i(g(a_i; x), y_i) + r(x)$$

with components
* $x \in \mathbf{R}^n$ - model parameters,
* $f_i: \mathbf{R}^n \to \mathbf{R}$ - functions that measure how well the model fits the data for a given set of parameters, and
* $r(x): \mathbf{R}^n \to \mathbf{R}$ - regularization function.

# Logistic Regression

In the logistic regression problem, we would like to find a linear predictor

$$g(a_i; x) = x_1 a_{i1} + \dots + x_n a_{in} = a_i^Tx \approx \begin{cases} \text{positive} & \Rightarrow \quad y_i = 1, \\ \text{negative} & \Rightarrow \quad y_i = 0, \end{cases}$$

where $a_i$ are continuous data features and $y_i \in \{0, 1\}$ are discrete class labels.
One approach for deriving the [loss functions](https://en.wikipedia.org/wiki/Loss_function) $f_i$ is to combine the [Bernoulli model](https://en.wikipedia.org/wiki/Bernoulli_distribution) with a linear predictor, and then develop a [maximum likelihood](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) formulation.
Let $p_i$ be the probability that observation $i$ belongs to class 1.
We begin by writing the probability mass function for an observation $(a_i, y_i)$ given model parameters $x$ in standard form:

\begin{align}
p((a_i, y_i); x) &= p_i^{y_i} (1 - p_i)^{1 - y_i} \\
&= \exp\big( y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \big) \\
&= \exp\left( y_i \log\left(\frac{p_i}{1 - p_i}\right) + \log(1 - p_i) \right)
\end{align}

Next, we assume that the term multiplied by $y_i$, the [log-odds](https://en.wikipedia.org/wiki/Logit) of observation $i$ belonging to class 1, can be approximated with our linear predictor:

$$\log\left(\frac{p_i}{1 - p_i}\right) \approx a_i^Tx$$

Substituting this into our probability above, the PMF of all $m$ i.i.d. observations is

\begin{align}
p\big(\{(a_1, y_1), \dots, (a_m, y_m)\}; x\big) &= \prod_{i=1}^m p((a_i, y_i); x) \\
&= \prod_{i=1}^m \exp\big(y_i a_i^Tx - \log\left(1 + \exp(a_i^Tx)\right) \big) \\
&= \exp\left(\sum_{i=1}^n y_i a_i^Tx - \log\left( 1 + \exp(a_i^Tx) \right) \right).
\end{align}

Alternatively, we can consider the likelihood of a set of model parameters given our $m$ observations,

$$\mathcal{L}\big(x; \{(a_1, y_1), \dots, (a_m, y_m) \} \big) = p\big(\{(a_1, y_1), \dots, (a_m, y_m)\}; x\big),$$

so that we can solve an optimization problem to find the parameters with the maximum likelihood.
In practice, this is often done by minimizing the negative log-likelihood, which results in our logistic regression problem

$$\min_x \sum_{i=1}^m \log\big( 1 + \exp(a_i^Tx) \big) - y_i a_i^T x.$$

After solving for the model parameters, we can predict the class labels of new data using the sign of the linear predictor,

$$g(a_i; x) = \begin{cases} 1 & \text{if} \quad a_i^Tx \geq 0, \\ 0 & \text{if} \quad a_i^T < 0, \end{cases}$$

or we can approximate the probability of an observation belonging to class 1,

$$P(y_i = 1 | a_i, x) = \frac{\exp(a_i^Tx)}{1 + \exp(a_i^Tx)} = \frac{1}{1 + \exp(-a_i^Tx)},$$

where the [logistic model](https://en.wikipedia.org/wiki/Logistic_function) follows from our linear approximation of the log-odds.

# Outlier Detection/Removal via Trimming

[Outliers](https://en.wikipedia.org/wiki/Outlier) in the training data can sometimes make it difficult to find a good predictor.
These problematic points can come from many causes, such as malfunctioning sensors, fraudulent behavior, natural population variation, or even erroneus model assumptions.
One way to identify and remove outliers during the optimization proces is to add auxiliary weights to each training example, called [trimming](https://en.wikipedia.org/wiki/Least_trimmed_squares):

$$min_{x, w} \sum_{i=1}^m w_k f_i(x) \quad \text{s.t.} \quad w_i \in [0, 1], \quad \sum_{i=1}^m w_i = h$$

For a fixed $x$, the $h$ training examples with the smallest residuals will be classified as inliers ($w_i = 1$), while the remaining $m - h$ points will be classified as outliers ($w_i = 0)$.
Because the outliers have small weights, they do not influence the final solution.
Here $h$ is a hyperparameter representing the the number of inliers we expect in the training data.

This problem is now nonconvex, but we can still find a good solution using variations of the optimization methods we've seen before.
Here, our solver alternates between updating $x$ using a step of [Newton's method](https://en.wikipedia.org/wiki/Newton%27s_method_in_optimization) and updating $w$ using a step of [projected gradient descent](https://en.wikipedia.org/wiki/Proximal_gradient_methods_for_learning).

# Exercise 1: Simple Example

In the example below, experiment with the noise_level and inlier_pct variables to answer the following questions:
* Let inlier_pct = 1.0. Using noise_level = 0, 0.1, 0.25, and 0.5, how does the presence of outliers affect the shape of the classifier and its ability to predict class labels?
* Let noise_level = 0.25. Using inlier_pct = 1.0, 0.98, 0.96, 0.94, 0.92, and 0.9. How does the number of outliers detected affect the shape of the classifier and its ability to predict class labels?
* Play around with other parameter values to see how the presence of outliers and the ability to detect them changes the quality of the classifier.

```python
noise_level = 0; inlier_pct = 1.0 # modify these variables

# Generate data
n = 50
x = np.array([-10, 2])
A = np.array([np.ones(n), np.linspace(0, 10, n)]).T
p = lambda a: 1/(1 + np.exp(-(2*a - 10)))
np.random.seed(0)
y = np.array([0 if p(ii) + noise_level*np.random.standard_t(1) < 0.5 else 1 for ii in A[:, 1]])

# Create model
training_data = utils.ImageData(A, (2,), labels=y)
model = utils.BinaryLogisticRegression(training_data, lam=0.001, inlier_pct=inlier_pct)
classifier, outliers = model.fit_model()
x = classifier.classifier
if inlier_pct < 1:
    a2 = np.array([a[1] for a in outliers.images])
    y2 = np.array([0 if y == -1 else 1 for y in outliers.labels])
utils.print_blr(A, y, x, 'training')
print(f'True x: {[-10, 2]}, Classifier: {x}')
utils.plot_blr(A[:, 1], y, x, outliers)
```

# Exercise 2: MNIST

The [MNIST](http://yann.lecun.com/exdb/mnist/) data set contains images of handwritten digits.
Each image is 28x28 pixels, which we reshape into feature vectors $a_i \in \mathbf{R}^{784}$.
For our examples, we will consider a subset of 1000 training images and 200 test images that contain zeros and ones, and our goal will be to build a classifier to correctly predict whether a given image contains a zero or a one.

```python
# Load data
A_train = np.load('../data/train_images.npy')
y_train = np.load('../data/train_labels.npy')
A_test = np.load('../data/test_images.npy')
y_test = np.load('../data/test_labels.npy')
print(f'Number of training images: {y_train.size}')
print(f'Number of testing images: {y_test.size}')

# Show two examples
fig, ax = plt.subplots(1, 2)
ax[0].imshow(A_train[1].reshape(28, 28))
ax[0].axis('off')
ax[1].imshow(A_test[2].reshape(28, 28))
ax[1].axis('off');
```

We will build a classifier for the MNIST 0-1 images using logistic regression with 2-norm regularization by solving the problem

$$\min_x \sum_{i=1}^m \log\big( 1 + \exp(a_i^Tx) \big) - y_i a_i^T x + \frac{\lambda}{2} \| x \|_2^2,$$

where $a_i \in \mathbf{R}^{784}$, $y_i \in \{0, 1\}$, and $x \in \mathbf{R}^{784}$.
Once we have solved for $x$, we can classify an image as a zero or a one based on the sign of $a_i^Tx$.

```python
lam = 0.1
step = 1/(np.linalg.norm(A_train)**2/4 + lam)
def func(x):
    z = A_train.dot(x)
    return np.sum(np.log(1 + np.exp(z)) - y_train*z) + lam*np.sum(x**2)/2
def grad(x):
    z = A_train.dot(x)
    return A_train.T.dot(np.exp(z)/(1 + np.exp(z)) - y_train) + lam*x
results = utils.gradient_descent(func, grad, step=step, x0=np.zeros_like(A_train[0]))
utils.print_blr(A_train, y_train, results[0], 'training')
utils.print_blr(A_test, y_test, results[0], 'test')
utils.plot_mnist(results)
```

Below, we see our solution vector $x^*$ reshaped into a 28x28 image.
To produce negative values for images containing a zero, the classifier is negative in regions where the zero images are likely to be positive.
Likewise, to produce positive values for images containing ones, the classifier is positive in regions where one images are likely to be positive.

```python
fig, ax = plt.subplots(1, 3, figsize=(20, 5))
f0 = ax[0].imshow(results[0].reshape(28, 28))
ax[0].axis('off')
fig.colorbar(f0, ax=ax[0])
f1 = ax[1].imshow(A_train[1].reshape(28, 28))
ax[1].axis('off')
fig.colorbar(f1, ax=ax[1])
f2 = ax[2].imshow(A_train[2].reshape(28, 28))
ax[2].axis('off')
fig.colorbar(f2, ax=ax[2]);
```

Using this classifier, we were able to correctly classify almost every image in the training and test data sets.
So what about the one image we could not correctly classify?
Below, we see that while this image contains a zero, it is much narrower than the negative regions in the classifier.
Let's see if we can use trimming for our regularized logistic regression model to identify other images that don't quite fit within the typical zero or one images.

```python
p_train = 1.0*(A_train.dot(results[0]) >= 0)
plt.imshow(A_train[p_train != y_train].reshape(28, 28))
plt.axis('off');
```

In the example below, experiment with the inlier_pct variable to answer the following questions:
* How does the number of outliers trimmed affect the number of correctly classified images in the training and test sets?
* How does the number of outliers trimmed affect the shape of the classifier?
* Looking at a sample of the outliers, can you guess why these images were trimmed?

```python
inlier_pct = 1.0 # modify this variable

# Get classifier
training_data = utils.ImageData(A_train, (28, 28), labels=y_train)
testing_data = utils.ImageData(A_test, (28,28), labels=y_test)
model = utils.BinaryLogisticRegression(training_data, lam=0.1, inlier_pct=inlier_pct)
classifier, outliers = model.fit_model()
utils.print_blr(A_train, y_train, classifier.classifier, 'train')
utils.print_blr(A_test, y_test, classifier.classifier, 'test')
classifier.plot_classifier()

# Plot outliers
k = 5
if outliers is not None:
    print(f'Number of outlier images: {outliers.num_images}')
    print(f'Number of \"0\" outlier images: {outliers.class_sizes[0]}')
    print(f'Number of \"1\" outlier images: {outliers.class_sizes[1]}')
    if outliers.num_images >= 2*k:
        fig, ax = plt.subplots(2, k, figsize=(15, 5))
        idx0 = np.argwhere(outliers.labels == -1)
        idx1 = np.argwhere(outliers.labels == 1)
        for ii in range(k):
            ax[0, ii].imshow(outliers.images[idx0[ii]].reshape(28, 28))
            ax[0, ii].axis('off')
            ax[1, ii].imshow(outliers.images[idx1[ii]].reshape(28, 28))
            ax[1, ii].axis('off')
```

# Further Reading
* Rousseeuw, P. J. (1985). "Multivariate estimation with high breakdown point." *Mathematical Statistics and Applications*, 8(283-297), 37.
* Neykov, N. M., & Müller, C. H. (2003). "Breakdown point and computation of trimmed likelihood estimators in generalized linear models." In *Developments in Robust Statistics* (pp. 277-286). Physica, Heidelberg.
* Hosmer Jr, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). *Applied logistic regression* (Vol. 398). John Wiley & Sons.
* Yang, E., Lozano, A. C., & Aravkin, A. (2018). "A general family of trimmed estimators for robust high-dimensional data analysis." *Electronic Journal of Statistics*, 12(2), 3519-3553.
* Aravkin, A., & Davis, D. (2020). "Trimmed statistical estimation via variance reduction." *Mathematics of Operations Research*, 45(1), 292-322.


# Eigenvalues (Lecture 5)

```python
# This cell just imports relevant modules

import numpy as np
# import pylab
from math import pi
from sympy import sin, cos, Function, Symbol, diff, integrate, matrices
# %matplotlib inline
import matplotlib.pyplot as plt
```

## Transformation matrices

**Slide 9**

```python
# Define the transformation matrix in Python using numpy.array
mD = np.array([[1.25, 0],
               [0, 0.8]])

# A list of coordinate vectors in the form [x,y].
# These are stored as numpy arrays so we can easily multiply them
# by the transformation matrix.
vCoordinates = [np.array([1, 0]),
               np.array([0, 1]),
               np.array([-1, 0]),
               np.array([0, -1])]

# Take each coordinate and transform it
for i in range(len(vCoordinates)):
    # We need to reshape the array so it is conformable
    # (i.e. it is of the right dimension for matrix-vector
    # multiplication). In this case, we need it to be 2 x 1.
    vCoordinates[i] = np.reshape(vCoordinates[i], (2,1))
    print(f"Transformed vCoordinates[{i}] = \n", mD @ vCoordinates[i])

sDet = np.linalg.det(mD)
sVolStrain = sDet - 1  # The volumetric strain

print("Determinant of D is: %f" % sDet)
print("This implies:")

if(sVolStrain == 1):
    print("No volume change")
elif(sVolStrain > 1):
    print("Increase in volume")
elif(sVolStrain > 0 and sVolStrain < 1):
    print("Decrease in volume")
else:
    print("No geological meaning")
```

    Transformed vCoordinates[0] =
     [[1.25]
     [0.  ]]
    Transformed vCoordinates[1] =
     [[0. ]
     [0.8]]
    Transformed vCoordinates[2] =
     [[-1.25]
     [ 0.  ]]
    Transformed vCoordinates[3] =
     [[ 0. ]
     [-0.8]]
    Determinant of D is: 1.000000
    This implies:
    No geological meaning

## Eigenvalues of a \\( 2 \times 2 \\) matrix

**Slide 18**

A simple way of finding eigenvalues is to use the `numpy.linalg.eigvals` function:

```python
mA = np.array([[1,4],
               [1,1]])

sEigValues = np.linalg.eigvals(mA)
print("The eigenvalues of mA are: ", sEigValues)
```

    The eigenvalues of mA are:  [ 3. -1.]

Alternatively, we could work out the characteristic polynomial for \\( \lambda \\) and then find the roots. We do that using the `np.poly(A)` function which computes \\( \det ( A - \lambda I ) \\) and returns a list of scalar coefficients `[X, Y, Z]` for the characteristic polynomial \\( X \lambda^2 + Y \lambda + Z \\)

```python
sCharacteristicPoly = np.poly(mA)
print("The characteristic polynomial of mA is: (%d*lambda**2) + (%d*lambda) + (%d)"
      % (sCharacteristicPoly[0], sCharacteristicPoly[1], sCharacteristicPoly[2]))

# Finds the roots (in this case, these will be the eigenvalues)
sRoots = np.roots(sCharacteristicPoly)
print("The roots of the characteristic polynomial are: ", sRoots)
```

    The characteristic polynomial of mA is: (1*lambda**2) + (-2*lambda) + (-2)
    The roots of the characteristic polynomial are:  [ 3. -1.]

## Eigenvectors of a \\(2 \times 2\\) matrix

**Slide 20**

A simple way of finding eigenvectors is to use the `numpy.linalg.eig` function. Note that this gives BOTH the eigenvalues in an array (say `sEigValues`) **AND** eigenvectors in a matrix data type (say `vEigVectors`).

```python
mA = np.array([[1,4],
               [1,1]])

# The i-th eigenvector, stored in the column vEigVectors[:,i],
# corresponds to the eigenvalue stored in sEigValues[i].
(sEigValues, vEigVectors) = np.linalg.eig(mA)
for i in range(0, len(sEigValues)):
    print("Eigenvalue #", str(i+1), "is:", sEigValues[i])
    print("The corresponding eigenvector is:\n", vEigVectors[:,i])
```

    Eigenvalue # 1 is: 3.0000000000000004
    The corresponding eigenvector is:
     [0.89442719 0.4472136 ]
    Eigenvalue # 2 is: -0.9999999999999996
    The corresponding eigenvector is:
     [-0.89442719  0.4472136 ]

### Plotting function

Let us plot these eigenvectors:

```python
def plot_ellipse(mM):  #, horizontal_scale, vertical_scale):

    (sEigValues, vEigVectors) = np.linalg.eig(mM)

    # points (eigenvectors) on unit circle
    sp1 = np.array([vEigVectors[0][0], vEigVectors[1][0]])
    sp2 = np.array([vEigVectors[0][1], vEigVectors[1][1]])

    # transformed points (eigenvectors)
    tsp1 = mM @ sp1
    tsp2 = mM @ sp2

    # ellipse
    x0 = 0  # x-position of center
    y0 = 0  # y-position of center
    a = abs(sEigValues[0])  # radius on x-axis
    b = abs(sEigValues[1])  # radius on y-axis
    t_rot = np.arctan(vEigVectors[1][0]/vEigVectors[0][0])  # rotation angle

    t = np.linspace(0, 2*pi, 100)

    Ell = np.array([a*np.cos(t), b*np.sin(t)])  # points on an ellipse
    R_rot = np.array([[cos(t_rot), -sin(t_rot)],[sin(t_rot), cos(t_rot)]])  # 2-D rotation matrix
    Ell_rot = np.zeros((2, Ell.shape[1]))
    for i in range(Ell.shape[1]):
        Ell_rot[:,i] = R_rot @ Ell[:,i]  # rotated ellipse coordinates

    plt.figure(figsize=(12,12))
    plt.plot([0, sp1[0]], [0, sp1[1]], 'orange', label=f'eigenvector 1 = [{vEigVectors[0][0]:.2f}, {vEigVectors[1][0]:.2f}]', linewidth=5)
    plt.plot([0, tsp1[0]], [0, tsp1[1]], 'k--', label='eigenvalue 1 = {:.0f}'.format(sEigValues[0]))
    plt.plot([sp1[0], tsp1[0]], [sp1[1], tsp1[1]], 'ro')

    plt.plot([0, sp2[0]], [0, sp2[1]], 'c', label=f'eigenvector 2 = [{vEigVectors[0][1]:.2f}, {vEigVectors[1][1]:.2f}]', linewidth=5)
    plt.plot([0, tsp2[0]], [0, tsp2[1]], 'b--', label='eigenvalue 2 = {:.0f}'.format(sEigValues[1]))
    plt.plot([sp2[0], tsp2[0]], [sp2[1], tsp2[1]], 'bo')

    plt.plot(np.cos(t), np.sin(t), 'k', label='unit circle')
    plt.plot(x0+Ell_rot[0,:], y0+Ell_rot[1,:], 'g', label='transformed unit circle')

    plt.legend(loc='best', fontsize=12)
    plt.grid(True)
    plt.gca().set_aspect('equal')
    plt.show()

    ang = np.allclose(sp1 @ sp2, 0)
    if ang == True:
        print("The eigenvectors are orthogonal")
    else:
        print("The eigenvectors are not orthogonal")
```

```python
M = np.array([[1, 4],
              [1, 1]])

# call the function we defined above
plot_ellipse(M)
```

## Repeated eigenvalues

**Slide 24**

```python
#NOTE: Here we could also define mA using numpy.identity(2).
mA = np.array([[1, 0],
               [0, 1]])

#NOTE: NumPy will give ALL eigenvalues, including the repeated ones.
(sEigValues, vEigVectors) = np.linalg.eig(mA)

for i in range(0, len(sEigValues)):
    print("Eigenvalue #", str(i+1), " is: ", sEigValues[i])
    print("The corresponding eigenvector is:\n", vEigVectors[:,i])
```

    Eigenvalue # 1  is:  1.0
    The corresponding eigenvector is:
     [1. 0.]
    Eigenvalue # 2  is:  1.0
    The corresponding eigenvector is:
     [0. 1.]

## Real and complex eigenvalues

**Slide 28**

NumPy prints out complex numbers in the form \\(c = a + bj\\), where \\(j\\) is the imaginary number. We can use `numpy.real(c)` and `numpy.imag(c)` to print the real and imaginary parts respectively.

```python
mA = np.array([[0, 1],
               [-1, 0]])

sEigValues = np.linalg.eigvals(mA)
print("The eigenvalues of A are:", sEigValues)
print("The real part of the first eigenvalue is:", np.real(sEigValues[0]))
print("The imaginary part of the first eigenvalue is:", np.imag(sEigValues[0]))
```

    The eigenvalues of A are: [0.+1.j 0.-1.j]
    The real part of the first eigenvalue is: 0.0
    The imaginary part of the first eigenvalue is: 1.0

## Example eigenvalue problem

**Slide 30**

```python
mM = np.array([[3, -1],
               [-1, 3]])

(sEigValues, vEigVectors) = np.linalg.eig(mA)
for i in range(0, len(sEigValues)):
    print("Eigenvalue #", i+1, "is:", sEigValues[i])
    print("The corresponding eigenvector is:\n", vEigVectors[:,i])
```

    Eigenvalue # 1 is: 1j
    The corresponding eigenvector is:
     [0.70710678+0.j         0.        +0.70710678j]
    Eigenvalue # 2 is: -1j
    The corresponding eigenvector is:
     [0.70710678-0.j         0.        -0.70710678j]

```python
M = np.array([[3, -1],
              [-1, 3]])

# call the function we defined above
plot_ellipse(M)
```

## Symmetric matrices

**Slide 31**

Unfortunately, because referencing a column of the matrix vEigVectors also returns another array data type (essentially a 'sub-array' of vEigVectors), we cannot use the `numpy.dot` function (which only operates on vectors/1D arrays). Instead, we'll simply use `numpy.transpose` to perform the dot product instead.

Note: We could also use [`numpy.vdot`](https://numpy.org/doc/stable/reference/generated/numpy.vdot.html) - read the documentation for more info.

```python
mNonSymmetric = np.array([[1, 4],
                          [1, 1]])

(sEigValues, vEigVectors) = np.linalg.eig(mNonSymmetric)

for i in range(0, len(sEigValues)):
    print("Eigenvalue #", i+1, "is:", sEigValues[i])
    print("The corresponding eigenvector is:\n", vEigVectors[:,i])

print("The dot product of the two eigenvectors is:", np.transpose(vEigVectors[:,0]) @ vEigVectors[:,1])
```

    Eigenvalue # 1 is: 3.0000000000000004
    The corresponding eigenvector is:
     [0.89442719 0.4472136 ]
    Eigenvalue # 2 is: -0.9999999999999996
    The corresponding eigenvector is:
     [-0.89442719  0.4472136 ]
    The dot product of the two eigenvectors is: -0.6000000000000001

Above, the dot product of the two eigenvectors was different from zero.

Below, the dot product should be zero as the two eigenvectors are orthogonal for any symmetric matrix.

```python
mSymmetric = np.array([[3, -1],
                       [-1, 3]])

(sEigValues, vEigVectors) = np.linalg.eig(mSymmetric)
for i in range(0, len(sEigValues)):
    print("Eigenvalue #", str(i+1), " is: ", sEigValues[i])
    print("The corresponding eigenvector is:\n", vEigVectors[:,i])

print("The dot product of the two eigenvectors is: ", np.transpose(vEigVectors[:,0]) @ vEigVectors[:,1])
```

    Eigenvalue # 1  is:  4.0
    The corresponding eigenvector is:
     [ 0.70710678 -0.70710678]
    Eigenvalue # 2  is:  2.0
    The corresponding eigenvector is:
     [0.70710678 0.70710678]
    The dot product of the two eigenvectors is:  0.0

## Eigenvalue problem for a \\(3 \times 3\\) matrix

**Slide 37**

```python
mM = np.array([[2, 2, 1],
               [1, 3, 1],
               [1, 2, 2]])

(sEigValues, vEigVectors) = np.linalg.eig(mM)

for i in range(0, len(sEigValues)):
    print("Eigenvalue #", i+1, " is: ", sEigValues[i])
    print("The corresponding eigenvector is:\n", vEigVectors[:,i])
```

    Eigenvalue # 1  is:  1.0
    The corresponding eigenvector is:
     [-0.90453403  0.30151134  0.30151134]
    Eigenvalue # 2  is:  4.999999999999998
    The corresponding eigenvector is:
     [0.57735027 0.57735027 0.57735027]
    Eigenvalue # 3  is:  0.9999999999999997
    The corresponding eigenvector is:
     [ 0.04093652 -0.46313831  0.88534011]

If the values of \\(x_1, x_2\\) or \\(x_3\\) are 'free' (i.e. can be chosen arbitrarily to obtain an independent eigenvector), NumPy does not necessarily choose values of 0 or 1, which is why the eigenvectors printed out here are different from those in your notes. They still satisfy the equation \\( \(A - \lambda I) \mathbf{x} = 0\\) and are still independent eigenvectors.

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import sparse, fftpack
from math import factorial as fact
try:
    plt.style.use("jupyter")
except OSerror:
    print("Using default ploting style")
# L-p norm function
norm = lambda v, p=2 : (len(v)**(-p)*np.einsum('i->', np.abs(v)**2))**(1./p)
```

### Compact n$^{th}$-order derivative

The compact coefficients for the $n^{th}$ derivative $f^{(n)}$ of a function can be found by solving the system

$$
\begin{bmatrix}
    \begin{matrix}
        0 & 0 & 0
    \end{matrix} & \begin{matrix}
        1 & 1 & 1\\
    \end{matrix}\\
    Q^{(n)} & \begin{matrix}
h_{i-1} & 0 & h_{i+1}\\
h_{i-1}^2/2! & 0 & h_{i+1}^2/2!\\
h_{i-1}^3/3! & 0 & h_{i+1}^3/3!\\
h_{i-1}^4/4! & 0 & h_{i+1}^4/4!
\end{matrix}\\
\begin{matrix}
        0 & 1 & 0
    \end{matrix} & \begin{matrix}
        0 & 0 & 0\\
    \end{matrix}\\
\end{bmatrix}\begin{bmatrix}
L_{i-1} \\ L_{i} \\ L_{i+1} \\ -R_{i-1} \\ -R_{i} \\ -R_{i+1}\\
\end{bmatrix}=\begin{bmatrix}
0\\ 0\\ 0\\ 0\\ 0\\ 1\\,
\end{bmatrix}
$$

where $h_{i-1}=x_{i-1}-x_i$ and $h_{i+1} = x_{i+1}-x_i$. The sub-matrix $Q^{(n)}$ depends on the derivative required. For the first derivative, we have

$$
Q^{(1)} = \begin{bmatrix}
1 & 1 & 1\\
h_{i-1} & 0 & h_{i+1}\\
h_{i-1}^2/2! & 0 & h_{i+1}^2/2!\\
h_{i-1}^3/3! & 0 & h_{i+1}^3/3!\\
\end{bmatrix}
$$

and for the second derivative

$$
Q^{(2)} = \begin{bmatrix}
0 & 0 & 0\\
1 & 1 & 1\\
h_{i-1} & 0 & h_{i+1}\\
h_{i-1}^2/2! & 0 & h_{i+1}^2/2!\\
\end{bmatrix}.
$$

```python
def get_compact_coeffs(n, hi):
    # assumes uniform grid
    h_i = -hi
    r = np.hstack((np.array([0 for i in range(5)]),1.))
    L = np.array([[0, 0, 0, 1, 1, 1],
                  [0, 0, 0, h_i, 0, hi],
                  [0, 0, 0, h_i**2/fact(2), 0, hi**2/fact(2)],
                  [0, 0, 0, h_i**3/fact(3), 0, hi**3/fact(3)],
                  [0, 0, 0, h_i**4/fact(4), 0, hi**4/fact(4)],
                  [0, 1, 0, 0, 0, 0]])
    insert = np.array([[1,           1,                1],
                       [h_i,         0,               hi],
                       [h_i**2/fact(2), 0, hi**2/fact(2)],
                       [h_i**3/fact(3), 0, hi**3/fact(3)]])
    L[n:5,:3] = insert[:-n+5,:]
    vec = np.round(np.linalg.solve(L, r), 8)
    return vec[:3], -vec[3:]
```

We can check that for a first derivative, we recover the standard Pade ($4^{th}$-order) [coefficients](https://github.com/marinlauber/my-numerical-recipes/blob/master/Compact-Schemes.ipynb), which are

$$
   L = \left[\frac{1}{4}, 1, \frac{1}{4}\right], \qquad R = \left[-\frac{3}{4}, 0., \frac{3}{4}\right]
$$

```python
pade = np.array([1./4., 1., 1./4., -3./4., 0., 3./4.])
np.allclose(np.hstack(get_compact_coeffs(1, 1)), pade)
```

    True

We can now write a function that, given a function $f$, on a uniform grid with spacing $dx$, return the $n^{th}$ derivative of that function. Because for each point we solve for the compact coefficients, we can in theory get compact schemes on non-uniform grid with the same accuracy. Here we will only focs on uniform grids.

```python
def derive_compact(n, f, dx):

    # get coeffs
    L, R = get_compact_coeffs(n, dx)

    # temp array
    sol = np.empty_like(f)

    # compact scheme on interior points
    sol[2:-2] = R[0]*f[1:-3] + R[1]*f[2:-2] + R[2]*f[3:-1]

    # boundary points
    sol[-2] = R[0]*f[-3] + R[1]*f[-2] + R[2]*f[-1]
    sol[-1] = R[0]*f[-2] + R[1]*f[-1] + R[2]*f[-0]
    sol[ 0] = R[0]*f[-1] + R[1]*f[ 0] + R[2]*f[ 1]
    sol[ 1] = R[0]*f[ 0] + R[1]*f[ 1] + R[2]*f[ 2]

    # build ugly matrix by hand
    A = sparse.diags(L,[-1,0,1],shape=(len(f),len(f))).toarray()
    # periodic BS's
    A[ 0,-1] = L[0]
    A[-1, 0] = L[2]

    return np.linalg.solve(A, sol)
```

We can then test the method on a known function, with known first and second derivaive. For simplicity, we will use trigonometric functions, which have well-behaved infinite derivatives.

$$
    f(x) = \sin(x), \,\, x\in[0, 2\pi]
$$

such that

$$
    \frac{d}{dx}f(x) = \cos(x), \quad \frac{d^2}{dx^2}f(x) = -\sin(x), \,\, x\in[0, 2\pi]
$$

```python
N = 128
x, dx = np.linspace(0, 2*np.pi, N, retstep=True, endpoint=False)
function = np.sin(x)

# first derivative
sol = derive_compact(1, function, dx)
print('First derivative L2 norm:  ', norm(sol-np.cos(x)))

# second derivative
sol = derive_compact(2, function, dx)
print('Second derivative L2 norm: ', norm(sol+np.sin(x)))
```

    First derivative L2 norm:   2.00356231982653e-09
    Second derivative L2 norm:  1.5119843767976088e-09

### Poisson Equation With Compact Schemes

We aim to solve the following one-dimensionnal Poisson equation with Dirichlet boudnary conditions

$$
\begin{split}
    -&\frac{d^2}{dx^2}u(x) = f(x), \quad a<x<b\\
    &u(a) = 0, \quad u(b) = 0\\
\end{split}
$$

where $a, b\in\mathbb{R}$, $u(x)$ is the unkown function and $f(x)$ is some given source function. We discretize the left side of the Poisson equaution ($u''_i$) using a compact finite difference scheme with fourth-order accuracy on a uniform grid with grid points being $x_i = a+ih, h=(b-a)/M, i=0, 1, 2,..., M$ where $M$ is a positive integer.

$$
\frac{1}{10}u''_{i-1} + u''_i + \frac{1}{10}u''_{i+1} = \frac{6}{5}\frac{u_{i+1} + 2u_i + u_{i-1}}{h^2},
$$

or in a more common form,

$$
u''_{i-1} + 10u''_i + u''_{i+1} = \frac{12}{h^2}\left(u_{i+1} + 2u_i + u_{i-1}\right).
$$

This results in the following tri-diagonal system

$$
    AU''= \frac{12}{h^2}BU,
$$

where $U'' = (u''_1,u''_2,...,u''_M)^\top$ and $U = (u_1,u_2,...,u_M)^\top\in \mathbb{R}^{M-1}$. The tri-diagonal matrix $A, B \in \mathbb{R}^{M-1\times M-1}$ are

$$
A = \begin{bmatrix}
10 & 1 & 0 &\dots & 0 & 0 \\
1 & 10 & 1 &\dots & 0 & 0 \\
0 & 1 & 10 &\dots & 0 & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & 0 & \dots & 10 & 1 \\
0 & 0 & 0 &\dots & 1 & 10 \\
\end{bmatrix}, \qquad B = \begin{bmatrix}
-2 & 1 & 0 &\dots & 0 & 0 \\
1 & -2 & 1 &\dots & 0 & 0 \\
0 & 1 & -2 &\dots & 0 & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
0 & 0 & 0 & \dots & -2 & 1 \\
0 & 0 & 0 &\dots & 1 & -2 \\
\end{bmatrix}.
$$

In addition, we have $-u''(x_i)=f(x_i), i=1,2,...,M-1$ i.e. $-U''=F$. We can re-write out system as

$$
   -\frac{12}{h^2}BU = AF,
$$

To obtaine the solution $U$, we simply need to solve the system.

```python
def build_AB(M, h):
    A = sparse.diags([1.,10.,1.],[-1,0,1],shape=(M, M)).toarray()
    B = sparse.diags([1.,-2.,1.],[-1,0,1],shape=(M, M)).toarray()
    # dont forget BC, here homogeneous Dirichlet
    B[ 0,:]=0.; B[ 0, 0]=1
    B[-1,:]=0.; B[-1,-1]=1
    return A, -12./h**2*B
```

In the first example, we consider the problem with homogeneous Dirichlet boundary conditions

$$
\begin{cases}
    -u''(x) = \pi^2\sin(\pi x), & 0 < x <2,\\
      u(0)=0, \quad u(2) = 0.
\end{cases}
$$

The exact solution is $u_e(x)=\sin(\pi x)$.

```python
def SolvePoissonCompact(f, h, M):
    u0 = np.zeros_like(f)
    A, B = build_AB(M, h)
    sigma = np.matmul(A, f)
    return np.linalg.solve(B, sigma)
```

```python
M = 64
x, h = np.linspace(0., 2., M, retstep=True, endpoint=True)
f = np.pi**2*np.sin(np.pi*x)
u_e = np.sin(np.pi*x)
u_num =  SolvePoissonCompact(f, h, M)
print(norm(u_num-u_e))
plt.plot(x, u_e, '-s')
plt.plot(x, u_num,'-o')
plt.xlabel(r"$x$");plt.ylabel(r"$u$")
# plt.savefig("figure_1.png", dpi=300);
```

Now with non-zero Dirichlet Boundry conditions

$$
\begin{cases}
    -u''(x) = 12e^{-x^2}(-x^2+1/2), & -8 < x <8,\\
      u(-8)=-8, \quad u(8) = 8.
\end{cases}
$$

The exact solution is $u_e(x)=3e^{-x^2}$. I. the numerical computation, we denote $U(x)=u(x)-x$ using change of variable. Applying tthe numerical algorithm, we now have

$$
\begin{cases}
    -U''(x) = 12e^{-x^2}(-x^2+1/2), & -8 < x <8,\\
      U(-8)=-0, \quad U(8) = 0.
\end{cases}
$$

and the approximate numerical solution at a grid point is found as $u(x) = U(x)=x$.

```python
M = 64
x, h = np.linspace(-8., 8., M, retstep=True, endpoint=True)
f = 12.*np.exp(-x**2)*(-x**2 + 0.5)
u_e = 3.*np.exp(-x**2)+x
u_num = SolvePoissonCompact(f, h, M)
print(norm(u_num-u_e))
plt.plot(x, u_e, '-s')
plt.plot(x, u_num+x,'-o')
plt.xlabel(r"$x$");plt.ylabel(r"$u$");
# plt.savefig("figure_2.png", dpi=300);
```

### Using Faste Fourier Transforms to Solve the Poisson Equation

We actually do not need ton inverte the system described earlier to get the solution, [see](https://www.sciencedirect.com/science/article/pii/S0898122116300761). We can use the Sine transform for $U\in\mathbb{R}^{M-1}$

$$
\begin{split}
    u_j &= \sum_{k=1}^{M-1}\hat{u}_k\sin\left(\frac{jk\pi}{M}\right), \,\, j=1,2,...,M-1,\\
    \hat{u_k} &= \frac{2}{M}\sum_{j=1}^{M-1}u_j\sin\left(\frac{ik\pi}{M}\right), \,\, j=1,2,...,M-1,
\end{split}
$$

from whcih we can approximate $u_{i+1}, u_{i-1}, u''_{i+1}, u''_{i-1}$ as

$$
\begin{align}
   u_{i+1}=\sum_{k=1}^{M-1}\hat{u}_k\sin\left(\frac{(i+1)k\pi}{M}\right),\qquad & u_{i-1} = \sum_{k=1}^{M-1}\hat{u}_k\sin\left(\frac{(i-1)k\pi}{M}\right)\\
   u''_{i} =\sum_{k=1}^{M-1}\hat{u}''_k\sin\left(\frac{ik\pi}{M}\right),\qquad & u''_{i+1} =\sum_{k=1}^{M-1}\hat{u}''_k\sin\left(\frac{(i+1)k\pi}{M}\right)\\
   u''_{i-1} =\sum_{k=1}^{M-1}\hat{u}''_k\sin\left(\frac{(i-1)k\pi}{M}\right). & \\
\end{align}
$$

Subsituting in the compact discretization of the Poisson equation gives,

$$
\sum_{k=1}^{M-1}\hat{u}''_k\left\{ \frac{1}{10}\sin\left(\frac{(i-1)k\pi}{M}\right) + \sin\left(\frac{ik\pi}{M}\right) + \frac{1}{10}\sin\left(\frac{(i+1)k\pi}{M}\right) \right\} =\frac{6}{5h^2}\sum_{k=1}^{M-1}\hat{u}_k\left\{ \sin\left(\frac{(i-1)k\pi}{M}\right) +\sin\left(\frac{(i+1)k\pi}{M}\right) - 2\sin\left(\frac{ik\pi}{M}\right) \right\}
$$

or, after rearranging

$$
\hat{u}_k = -\hat{u}''_k\left(\frac{24\sin^2\left(\frac{k\pi}{2M}\right)}{h^2}\right)^{-1}\left(\cos\left(\frac{k\pi}{M}\right)+5\right), \,\, k\in 1,2,..,M-1.
$$

In addition, we obtain $-u''_i = f_i \,(i=1,2,...,M-1)$. By the inverse Sine transform, we get to know $-\hat{u}''_k=\hat{f}_k \, (k=1,2,...,M-1)$, whci allows us to solve for $\hat{u}$

$$
\hat{u}_k = \hat{f}_k\left(\frac{24\sin^2\left(\frac{k\pi}{2M}\right)}{h^2}\right)^{-1}\left(\cos\left(\frac{k\pi}{M}\right)+5\right), \,\, k\in 1,2,..,M-1.
$$

> **_Note:_** We use a spectral method to solve the tri-diagonal system, this doesn't mean we solve it with spectral accuracy, here the modified wavenumber makes the spectral method the exact same accuracy as the compact scheme.

```python
def SolvePoissonSine(f, h, M):
    f_k = fftpack.dst(f, norm='ortho')
    k = np.arange(1,M+1)
    u_k = f_k*(24*np.sin(k*np.pi/(2*M))**2./h**2.)**(-1.)*(np.cos(np.pi*k/M)+5.)
    return fftpack.idst(u_k, norm='ortho')
```

```python
M = 64
x, h = np.linspace(-8, 8, M, retstep=True, endpoint=True)
f = 12.*np.exp(-x**2)*(-x**2 + 0.5)
u_e = 3.*np.exp(-x**2)+x
u_num = SolvePoissonSine(f, h, M)
print(norm(u_num-u_e))
plt.plot(x, u_num + x, '-o')
plt.plot(x, u_e, 's')
plt.xlabel(r"$x$");plt.ylabel(r"$u$");
# plt.savefig("figure_3.png", dpi=300);
```

### Order of Accuracy

```python
L2_com = []
L2_Sine = []
Resolutions = 2.**np.arange(4,9)
for N in Resolutions:
    x, h = np.linspace(0., 2., int(N), retstep=True, endpoint=True)
    f = np.pi**2*np.sin(np.pi*x)
    u_e = np.sin(np.pi*x)
    u_num = SolvePoissonCompact(f, h, int(N))
    error = norm(u_num-u_e)
    L2_com.append(error)
    u_num = SolvePoissonSine(f, h, int(N))
    error = norm(u_num-u_e)
    L2_Sine.append(error)
plt.loglog(Resolutions, np.array(L2_com), '--o', label='Compact Schemes')
plt.loglog(Resolutions, np.array(L2_Sine), ':s', label='Sine Transform')
plt.loglog(Resolutions, Resolutions**(-4), ':k', alpha=0.5, label=r"$4^{th}$-order")
plt.xlabel("Resolution (N)"); plt.ylabel(r"$L_2$-norm Error")
plt.legend();
# plt.savefig("figure_4.png", dpi=300);
```


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also http://splines.readthedocs.io/.

# Uniform Cubic Hermite Splines

TODO: Image for the 1D case (points and slopes)?

TODO: Image for the 2D case (points and tangent vectors)? Probably combine two 1D examples?

TODO: Hermite's two-point interpolation formula?

```python
%matplotlib inline
from IPython.display import display
```

```python
import sympy as sp
sp.init_printing(order='rev-lex')
```

```python
from utility import NamedExpression, NamedMatrix
```

```python
t = sp.symbols('t')
```

```python
coefficients = sp.Matrix(sp.symbols('abm:4')[::-1])
coefficients
```

```python
b_monomial = sp.Matrix([t**3, t**2, t, 1]).T
b_monomial
```

```python
p = b_monomial.dot(coefficients)
p
```

We want to provide the value of the polynomial at time $t_0 = 0$ and $t_1 = 1$ (start and end point).
We also want to provide the first derivative (a.k.a. velocity, a.k.a. tangent vector) at the start and at the end.

\begin{align}
\boldsymbol{x}_0 &= \boldsymbol{p}(0)\\
\boldsymbol{x}_1 &= \boldsymbol{p}(1)\\
\boldsymbol{\dot{x}}_0 &= \boldsymbol{p}'(0)\\
\boldsymbol{\dot{x}}_1 &= \boldsymbol{p}'(1)
\end{align}

The curve segment defined by these 4 values is sometimes called *Ferguson cubic*.

Velocity = Tangent Vector = First Derivative:

```python
velocity = p.diff(t)
velocity
```

```python
x0 = NamedExpression('xbm0', p.subs(t, 0))
x1 = NamedExpression('xbm1', p.subs(t, 1))
xd0 = NamedExpression('xdotbm0', velocity.subs(t, 0))
xd1 = NamedExpression('xdotbm1', velocity.subs(t, 1))
```

```python
display(x0, x1, xd0, xd1)
```

Given an input vector of control values ...

```python
control_values_H = NamedMatrix(sp.Matrix([x0.name, x1.name, xd0.name, xd1.name]))
control_values_H.name
```

... we want to find a way to transform those into the coefficients of our cubic polynomial.

```python
M_H = NamedMatrix(r'{M_\text{H}}', 4, 4)
```

```python
coefficients_H = NamedMatrix(coefficients, M_H.name * control_values_H.name)
coefficients_H
```

And the other way round:

```python
control_values_H.expr = M_H.name.I * coefficients
control_values_H
```

```python
substitutions = x0, x1, xd0, xd1
```

```python
control_values_H.subs(substitutions)
```

```python
M_H.I = sp.Matrix([[expr.coeff(cv) for cv in coefficients]
                   for expr in control_values_H.subs(substitutions).name])
M_H.I
```

```python
print(_.expr)
```

This transforms the coefficients into our control values, but we need it the other way round:

```python
M_H
```

```python
print(_.expr)
```

Those are called Hermite basis functions:

```python
b_H = NamedMatrix(r'{b_\text{H}}', b_monomial * M_H.expr)
b_H.factor().simplify().T
```

```python
sp.plot(*b_H.expr, (t, 0, 1));
```

TODO: describe properties of Hermite basis functions

## Plotting

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
def hermite_curve(control_values):
    coeffs = sp.lambdify([], M_H.expr)() @ control_values

    def evaluate(t):
        t = np.expand_dims(t, -1)
        return t**[3, 2, 1, 0] @ coeffs

    return evaluate
```

```python
t_values = np.linspace(0, 1, 15)
```

```python
def plot_hermite(control_values, t_values, ax=None):
    if ax is None:
        ax = plt.gca()
    control_values = np.asarray(control_values)
    c = hermite_curve(control_values)
    xyuv = np.column_stack(np.vsplit(control_values, 2))
    # Move beginning of second arrow:
    xyuv[1, :2] -= xyuv[1, 2:]
    ax.quiver(*xyuv.T, angles='xy', scale_units='xy', scale=1, color='lightgrey')
    ax.plot(*c(t_values).T, '.')
    ax.axis('equal')
```

```python
plot_hermite([[0, 0], [5, 2], [1, 2], [1, -2]], t_values)
plot_hermite([[5, 2], [6, 1], [1, -2], [5, 1]], t_values)
```

TODO: same length, different angles?

```python
plot_hermite([[0, 0], [1, 0], [1, 1], [1, -1]], t_values)
plot_hermite([[0, 0], [1, 0], [0, 1], [0, -1]], t_values)
```

## Relation to Bézier Splines

Above, we were using two positions (start and end) and two tangent vectors as control values:

```python
coefficients_H
```

What about using four positions instead?

Let's use the point $\boldsymbol{\hat{x}}_0$ as "handle" connected to $\boldsymbol{x}_0$.
Same for $\boldsymbol{\hat{x}}_1$ and $\boldsymbol{x}_1$.

And since the tangents looked unwieldily long in the examples before, let's make the handles only a third of the length of the tangents:

```python
x_hat0 = NamedExpression('xhatbm0', x0.name + xd0.name / 3)
x_hat1 = NamedExpression('xhatbm1', x1.name - xd1.name / 3)
```

```python
control_values_B = NamedMatrix(
    sp.Matrix([x0.name, x_hat0.name, x_hat1.name, x1.name]),
    sp.Matrix([x0.name, x_hat0.expr, x_hat1.expr, x1.name]))
control_values_B
```

```python
M_HtoB = NamedMatrix(r'{M_\text{H$\to$B}}', 4, 4)
```

```python
sp.Eq(control_values_B.name, M_HtoB.name * control_values_H.name)
```

```python
M_HtoB.expr = sp.Matrix([[expr.coeff(cv) for cv in control_values_H.name]
                         for expr in control_values_B.expr])
M_HtoB.pull_out(sp.S.One / 3)
```

```python
print(_.expr)
```

```python
M_BtoH = NamedMatrix(r'{M_\text{B$\to$H}}', M_HtoB.I.expr)
M_BtoH
```

```python
print(_.expr)
```

```python
M_B = NamedMatrix(r'{M_\text{B}}', M_H.name * M_BtoH.name)
M_B
```

```python
M_B = M_B.subs([M_H, M_BtoH])
M_B
```

```python
b_B = NamedMatrix(r'{b_\text{B}}', b_monomial * M_B.expr)
b_B.T
```

```python
sp.plot(*b_B.expr, (t, 0, 1));
```

Those happen to be the cubic Bernstein polynomials and it turns out that we just invented Bézier curves!
See [separate notebook](bezier.ipynb) for more about them.