# Sensitivity indices for Sobol's G function

**Leif Rune Hellevik**, [leif.r.hellevik@ntnu.no](mailto:leif.r.hellevik@ntnu.no)

In this notebook we
illustrate how sensitivity indices may be computed
for Sobol's G function
[Archer et al. 1997](https://www.tandfonline.com/doi/abs/10.1080/00949659708811825). We will demonstrate how both
Monte Carlo methods and
polynomial chaos expansions may be used to
estimate both first order indices and
total indices.

The G function was chosen as an example for two reasons:

* The sensitivity indices have analytical solutions [Saltelli et al. 2010](https://www.sciencedirect.com/user/chooseorg?targetURL=%2Fscience%2Farticle%2Fpii%2FS0010465509003087).

* The G function can be used to generate test cases over a wide spectrum of difficulties

The notebook has four sections; in the first section the G
function is
presented along with the analytical expressions for the sensitivity
indices. The second and third sections demonstrate how polynomial
chaos
expansions and Monte Carlo methods may be used to approximate
the sensitivity
indices. The final section is devoted for comparison
of these two numerical
approaches.

For all sections we make use of the interactive features of the
notebooks, which allow you to experiment with how the values of the G
function
parameters and sample size influence the sensitivity indices
for the G function.
The intention is to let you gain understanding,
experience and intuition in a
efficient and convenient manner.

Run the first cell to initialise plotting and
printing modules for
later use (and system settings).

```
# ipython magic
%matplotlib notebook
%load_ext autoreload
%autoreload 2
import os, sys, inspect
# Use this if you want to include modules from a subfolder
cmd_subfolder = os.path.realpath(os.path.abspath(os.path.join(os.path.split(inspect.getfile( inspect.currentframe() ))[0],"python_source")))
if cmd_subfolder not in sys.path:
     sys.path.insert(0, cmd_subfolder)

%run matplotlib_header

import matplotlib.pyplot as plt
from present_output import print_vectors_relerror, print_3vectors_relerror
```

## Analytical computation of sensitivity indices for Sobol's G function

A function which has proved to be useful as a test
function with analytical solutions for the sensitivity indices is Sobol's G
function which is defined as:

$$
\begin{equation}
Y=G(X) =  G(X_1, X_2,\ldots,X_k,a_1, a_2,\ldots,a_k)  =
\prod_{i=1}^{k} g_i \label{eq:1} \tag{1}
\end{equation}
$$

where

$$
\begin{equation}
g_i = \frac{|{4 \, X_i}-2|+{a}_i}{1+{a}_i} \label{eq:2} \tag{2}
\end{equation}
$$

The input factors $X_i$ are assumed to be uniformly
distributed in the
interval $[0,1]$ with positive real-number coefficients $a_i$ $(a_i \geq 0).$  The number of
factors *k* can be varied as the reader
pleases, although the minimum
number to produce a meaningful inference is set at
three.

As you will explore below, the sensitivity $S_i$ of $G(X)$ in
([1](#eq:1)) with respect to a specific input factor $X_i$, will depend
on the
value of the corresponding coefficient $a_i$; small values of
$a_i$ (e.g.
$a_i=0$) will yield a high corresponding $S_i$, meaning
that $X_i$ is an
important/influential variable on the variance or
uncertainty of $G(X)$.

We have
implemented Sobol's G function in ([1](#eq:1)) and ([2](#eq:2))
in the code
snippet below:

```
# model function
import numpy as np
from numba import jit

@jit
def g(Xj,aj):
    return (np.abs(4*Xj-2.)+aj)/(1+aj)

@jit
def G(X,a):
    G_vector=np.ones(X.shape[0])

    for j, aj in enumerate(a):
        np.multiply(G_vector,g(X[:,j],aj),G_vector)
    return G_vector
```

For computational efficiency we make use of `just in time` compilation
from
`numba`. If you have not installed `numba`, you
may comment out
the lines with `@jit` - the cells will run anyway,
albeit probably somewhat slower.

The sensitivity indices $S_i$ and $S_{Ti}$ for $Y=G(X)$ in
eq.
([1](#eq:1)) may be derived as outlined in [Saltelli et al. 2010](https://www.sciencedirect.com/science/article/pii/S0010465509003087).
The conditional variance $V_i$ is:

$$
\begin{equation}
V_i = V_{X_i} \left (E_{X_{\sim i}} (Y \;| \;X_{i}) \right) =
\frac{1/3}{(1+a_i)^2} \label{eq:Vi} \tag{3}
\end{equation}
$$

while the
$V_{T_I}$ and the total variance $V$ are given by:

$$
\begin{equation}
V_{T_i} = V_i \; \prod_{j\neq i} (1+V_j) \qquad \text{and}
\qquad V = \prod_{i=1}^k (1+V_i) -1
\label{eq:4} \tag{4}
\end{equation}
$$

Consequently the first order sensitivity indices $S_i$ of $Y=G(X)$, are given by

$$
\begin{equation}
S_i=\frac{V_i}{V}
\label{_auto1} \tag{5}
\end{equation}
$$

In the code snippet below one can
interactively experiment how the values of $a_i$ affect the correspoding
$S_i$, i.e the
sensitivity of $G$ with respect to $X_i$.

```
# Analytical computations
f, ax = plt.subplots(1,1)
f.suptitle('G function variable a coefficients')

# import modules
import numpy as np

def Vi(ai):
    return 1/(3*(1+ai)**2)

def V(a_prms):
    D=1
    for a in a_prms:
        D*=(1+Vi(a))
    return D-1

def S_i(ai,a):
    return Vi(ai)/V(a)

def S_T(ai,a):
    Dtot=V(a)
    return (Dtot+1)/(Vi(ai)+1)*Vi(ai)/Dtot

def update_Sobol(**kwargs):
    ax.clear()
    for key, value in kwargs.items(): #find indx and value for a_prms
        pre,post = key.split("a")
        assert pre==""
        a_prms[int(post)] = value

    width=0.4
    x_tick_list=np.arange(len(a_prms))+1
    ax.set_xticks(x_tick_list+width/2)
    x_labels=['x'+str(i) for i in np.arange(len(a_prms))]
    ax.set_xticklabels(x_labels)
    ax.set_ylim(0,1)

    for i, a in enumerate(a_prms):
        Si[i]=S_i(a,a_prms)
        ST[i]=S_T(a,a_prms)

    ax.bar(x_tick_list,Si,width,color='blue')
    ax.bar(x_tick_list+width,ST,width,color='red')
    ax.legend(['First order indices','Total indices'])

# Analytical sliders
k=4 #number of prms
a_lbls=['a'+str(i) for i in np.arange(k)]
Si=np.zeros(k)
ST=np.zeros(k)
a_prms=np.zeros(k)

import ipywidgets as widgets
my_sliders=[]
for i in range(k):
    my_sliders.append(widgets.FloatSlider(min=0, max=15, value=6.52, description=a_lbls[i]))

slider_dict = {slider.description:slider for slider in my_sliders}
ui_left = widgets.VBox(my_sliders[0::2])
ui_right = widgets.VBox(my_sliders[1::2])
ui=widgets.HBox([ui_left,ui_right])

out=widgets.interactive_output(update_Sobol, slider_dict)
display(ui,out)
```

Do you observe the effect stated above, that small
values of
$a_i$ (e.g. $a_i=0$) will yield high corresponding $S_i$? You may
change all the parameters simultaneously or one at the time.

You may also
change the number of parameters *k* directly in the
python-code, however note
that this will affect the
computing time. In particular, the
computing time for the numerical
approximations with *chaospy* will be sensitive
to *k*.

If more than one factor has low $a_i$, high order interactions among factors will
be tangible.

* The extreme case is setting all $a_i$’s to zero. In this circumstance, all factors will interact and will be of equal importance -> check it out!

* How would you assess a setting with only some $a_i$’s are zero and all others are large (e.g. $a_i \geq 9$ )?

Note that the G function has a singularity in each of its $k$ dimensions corresponding to the points $X_i = 1/2$.

## Approximation of the sensitivity indices for Sobol's G function with spectral expansions

In this section we show how the spectral expansion module [chaospy](https://github.com/jonathf/chaospy) may
be used to compute the Sobol indices for Sobol's G function.  A more in-depth treatment of
`chaospy` and its usage is provided in the separate notebook [A
practical introduction to polynomial chaos with the1 `chaospy`
package](introduction_gpc.ipynb). Furthermore, you may find our previous "A
Guide to Uncertainty Quantification and Sensitivity Analysis for
Cardiovascular
Applications" [Eck et al. 2015](https://onlinelibrary.wiley.com/doi/full/10.1002/cnm.2755) as a useful
introduction to how polynomial chaos expansions may be used for
UQ&S. We are therefore focusing on
the application of the spectral
expansions and how this benchmarks against the
analytical solutions for the
indices, rather than presenting the spectral
expansion theory.

```
# Si with chaospy for G-function
import chaospy as cp
cp.seed(0)
jpdf = cp.Iid(cp.Uniform(),k)

polynomial_order = 4
poly = cp.orth_ttr(polynomial_order, jpdf)

#Ns=2*len(poly)
Ns=500
print('Number of samples for chaospy: ', Ns)
X=jpdf.sample(Ns)
G_sample=G(X.transpose(),a_prms)

approx = cp.fit_regression(poly, X, G_sample)

exp_pc = cp.E(approx, jpdf)
std_pc = cp.Std(approx, jpdf)
print("Statistics polynomial chaos\n")
print('\n        E(Y)  |  std(Y) \n')
print('pc  : {:2.5f} | {:2.5f}'.format(float(exp_pc), std_pc))
S_pc = cp.Sens_m(approx, jpdf) #Si from chaospy
S_tpc = cp.Sens_t(approx, jpdf) #Total effect sensitivity index from chaospy
row_labels= ['S_'+str(idx) for idx in range(k)]
col_labels=['Chaospy','Analytical','Error (%)']

print("\nFirst Order Indices")

print_vectors_relerror(S_pc,Si,col_labels,row_labels,[3,3,0])

print("\n\nTotal Effect Indices")
row_labels= ['St_'+str(idx) for idx in range(k)]
print_vectors_relerror(S_tpc,ST,col_labels,row_labels,[3,3,0])
```

In the code-snippet above we compare both the first order indices `S_pc` and the
total indices `S_tpc` computed with chaospy, and print them in columns along
side the analytical indices and the relative errors. You may experiment how the
error is affected by the number of samples `Ns`.

### Spectral expansions for computation of  Sobol's sensitivity indices

To better facilitate and encourage
your experimentation with the impact of changes in the coefficients $a_i$,
number of samples, and polynomial order for the spectral expansions in the
chaospy module, we make use of interactive widgets with sliders for all these
coefficients and variables.

Run the code snippet below, and you will see
sliders for $a_i$, number
of samples `NS` and polynomial order. Once you change
one of the slider
values, the chaospy approximations of the sensitivity indices
is recomputed and the new results will be presented. Bear in mind that
the computational time is dependent on the number of samples and the
_cpu_ capacity of the machine you are running this notebook on.

```
# chaospy G-function with sliders
import chaospy as cp

if not 'jpdf' in globals():
    jpdf = cp.Iid(cp.Uniform(),k) #the joint pdf
    print('Create the joint pdf')

def update_chaospy_G(**kwargs):
    NS=kwargs['NS']
    del kwargs['NS']
    polynomial_order=kwargs['polynomial_order']
    del kwargs['polynomial_order']

    for key, value in kwargs.items(): #find indx and value for a_prms
        pre,post = key.split("a")
        assert pre==""
        a_prms[int(post)] = value

    X=jpdf.sample(NS)
    print('Number of samples: ',NS)

    G_sample=G(X.transpose(),a_prms)

    poly = cp.orth_ttr(polynomial_order, jpdf)
    approx = cp.fit_regression(poly, X, G_sample)

    exp_pc = cp.E(approx, jpdf)
    std_pc = cp.Std(approx, jpdf)
    print("Statistics polynomial chaos\n")
    print('\n        E(Y)  |  std(Y) \n')
    print('pc  : {:2.5f} | {:2.5f}'.format(float(exp_pc), std_pc))
    S_pc = cp.Sens_m(approx, jpdf) #Si from chaospy
    S_tpc = cp.Sens_t(approx, jpdf) #Total effect sensitivity index from chaospy

    row_labels= ['S_'+str(idx) for idx in range(len(a_prms))]
    col_labels=['Chaospy','Analytical','Error (%)']

    print("\nFirst Order Indices")
    print_vectors_relerror(S_pc,Si,col_labels,row_labels,[3,3,0])

    print("\n\nTotal Effect Indices")
    row_labels= ['St_'+str(idx) for idx in range(k)]
    print_vectors_relerror(S_tpc,ST,col_labels,row_labels,[3,3,0])

if (len(my_sliders)==len(a_prms)):   #add sliders if not added before
    my_sliders.append(widgets.IntSlider(min=500,max=5100,step=250,value=500,description='NS')) #add slider for samples
    my_sliders.append(widgets.IntSlider(description='polynomial_order', min=1,max=6,value=4)) # add slider for polynomial order

    slider_dict = {slider.description:slider for slider in my_sliders} #add the sliders in the dictionary

    ui_left = widgets.VBox(my_sliders[0::2])
    ui_right = widgets.VBox(my_sliders[1::2])
    ui=widgets.HBox([ui_left,ui_right])

out=widgets.interactive_output(update_chaospy_G, slider_dict)
display(ui,out)

# end chaospy G-function with sliders
```

### Monte Carlo simulations for computation of sensitivity indices

The snippet of code below allows to evaluate the Sobol sensitivity indices for the same $G$ function in a Monte Carlo simulation. In analogy with the previous example, one can again vary the $a_i$ coefficients along with the number of runs of the simulation.

```
# Si with monte carlo for G-function

import monte_carlo as mc
a_prms=np.ones(k)

if not 'jpdf' in globals():
    cp.seed(0)
    jpdf = cp.Iid(cp.Uniform(),k) #the joint pdf
    print('Create the joint pdf')

def update_mc_G(**kwargs):
    Ns=kwargs['NS']
    del kwargs['NS']

    for key, value in kwargs.items(): #find indx and value for a_prms
        pre,post = key.split("a")
        assert pre==""
        a_prms[int(post)] = value

    print('Number of samples for Monte Carlo: ', Ns)
    X=jpdf.sample(Ns)
    A, B, C = mc.generate_sample_matrices_mc(Ns, k, jpdf, sample_method='R') #A, B, C already transposed
    G_A_sample = G(A, a_prms)
    G_B_sample = G(B, a_prms)
    G_C_sample_list = np.array([G(C_i, a_prms) for C_i in C]).T

    exp_mc = np.mean(G_A_sample)
    std_mc = np.std(G_A_sample)
    print("Statistics Monte Carlo\n")
    print('\n        E(Y)  |  std(Y) \n')
    print('mc  : {:2.5f} | {:2.5f}'.format(float(exp_mc), std_mc))

    S_mc, S_tmc = mc.calculate_sensitivity_indices_mc(G_A_sample, G_B_sample, G_C_sample_list)
    row_labels= ['S_'+str(idx) for idx in range(k)]
    col_labels=['Monte carlo','Analytical','Error (%)']

    print("\nFirst Order Indices")
    import analytical_g_function as agf

    Si=np.zeros(k)
    ST=np.zeros(k)
    for i, a in enumerate(a_prms):
        Si[i]=agf.S_i(a,a_prms)
        ST[i]=agf.S_T(a,a_prms)

    print_vectors_relerror(S_mc, Si, col_labels, row_labels, [3,3,0])

    print("\n\nTotal Effect Indices")
    row_labels= ['St_'+str(idx) for idx in range(k)]
    print_vectors_relerror(S_tmc, ST, col_labels, row_labels, [3,3,0])

## Set up the sliders
mc_sliders=[]
for i in range(k):
    mc_sliders.append(widgets.FloatSlider(min=0, max=15, value=6.52, description=a_lbls[i]))

mc_sliders.append(widgets.IntSlider(min=500,max=25000,step=250,value=500,description='NS')) #add slider for samples

slider_dict = {slider.description:slider for slider in mc_sliders} #add the sliders in the dictionary

ui_left = widgets.VBox(mc_sliders[0::2])
ui_right = widgets.VBox(mc_sliders[1::2])
ui=widgets.HBox([ui_left,ui_right])

out=widgets.interactive_output(update_mc_G, slider_dict)
display(ui,out)
```

### Comparison of MC and PC for sensitivity indices computations

Finally, the performance of the two approaches can be compared by benchmarking against the analytical values of the indices. Which approach performs better? Under which combination of $a_i$ coefficients? How many runs are required in order to get an error below a given threshold (e.g. 5%, 1%)?

```
# Si comparison of mc and pc for G-function

import monte_carlo as mc
a_prms=np.ones(k)

if not 'jpdf' in globals():
    cp.seed(0)
    jpdf = cp.Iid(cp.Uniform(),k) #the joint pdf
    print('Create the joint pdf')

def update_cmp(**kwargs):
    NsMC=kwargs['NsMC']
    del kwargs['NsMC']

    NsPC=kwargs['NsPC']
    del kwargs['NsPC']

    for key, value in kwargs.items(): #find indx and value for a_prms
        pre,post = key.split("a")
        assert pre==""
        a_prms[int(post)] = value

    ## Monte Carlo update
    print('Number of samples for Monte Carlo: ', NsMC)
    X_mc=jpdf.sample(NsMC)
    A, B, C = mc.generate_sample_matrices_mc(NsMC, k, jpdf, sample_method='R') #A, B, C already transposed
    G_A_sample = G(A, a_prms)
    G_B_sample = G(B, a_prms)
    G_C_sample_list = np.array([G(C_i, a_prms) for C_i in C]).T

    exp_mc = np.mean(G_A_sample)
    std_mc = np.std(G_A_sample)
    print("Statistics Monte Carlo\n")
    print('\n        E(Y)  |  std(Y) \n')
    print('mc  : {:2.5f} | {:2.5f}'.format(float(exp_mc), std_mc))

    S_mc, S_tmc = mc.calculate_sensitivity_indices_mc(G_A_sample, G_B_sample, G_C_sample_list)

    ## PC update
    Xpc=jpdf.sample(NsPC)
    G_sample=G(Xpc.transpose(),a_prms)

    approx = cp.fit_regression(poly, Xpc, G_sample)

    exp_pc = cp.E(approx, jpdf)
    std_pc = cp.Std(approx, jpdf)
    print("Statistics polynomial chaos\n")
    print('\n        E(Y)  |  std(Y) \n')
    print('pc  : {:2.5f} | {:2.5f}'.format(float(exp_pc), std_pc))
    S_pc = cp.Sens_m(approx, jpdf) #Si from chaospy
    S_tpc = cp.Sens_t(approx, jpdf) #Total effect sensitivity index from chaospy

    import analytical_g_function as agf
    Si=np.zeros(k)
    ST=np.zeros(k)
    for i, a in enumerate(a_prms):
        Si[i]=agf.S_i(a,a_prms)
        ST[i]=agf.S_T(a,a_prms)

    row_labels= ['S_'+str(idx) for idx in range(k)]
    col_labels=['Monte Carlo','Err (%)','PolyChaos','Err (%)']

    print("\nFirst Order Indices")
    print_3vectors_relerror(S_mc,S_pc, Si, col_labels, row_labels, [3,0,3,0])

    print("\n\nTotal Effect Indices")
    row_labels= ['St_'+str(idx) for idx in range(k)]
    print_3vectors_relerror(S_tmc,S_tpc, ST, col_labels, row_labels, [3,0,3,0])

## Set up the sliders
cmp_sliders=[]
for i in range(k):
    cmp_sliders.append(widgets.FloatSlider(min=0, max=15, value=6.52, description=a_lbls[i]))

cmp_sliders.append(widgets.IntSlider(min=500,max=100000,step=250,value=500,description='NsMC')) #slider for MC samples
cmp_sliders.append(widgets.IntSlider(min=500,max=2000,step=250,value=500,description='NsPC')) #slider for PC samples

slider_dict = {slider.description:slider for slider in cmp_sliders} #add the sliders in the dictionary

ui_left = widgets.VBox(cmp_sliders[0::2])
ui_right = widgets.VBox(cmp_sliders[1::2])
ui=widgets.HBox([ui_left,ui_right])

out=widgets.interactive_output(update_cmp, slider_dict)
display(ui,out)
```


# Is Julia fast?

Julia isn't fast *per se*.

One can write terribly slow code in any language, including Julia.

So let's ask a different question.

# *Can* Julia be fast?

 ### Microbenchmarks

### More realistic case: Vandermonde matrix
(modified from [Steve's Julia intro](https://web.mit.edu/18.06/www/Fall17/1806/julia/Julia-intro.pdf))

[Vandermonde matrix:](https://en.wikipedia.org/wiki/Vandermonde_matrix)
\begin{align}V=\begin{bmatrix}1&\alpha _{1}&\alpha _{1}^{2}&\dots &\alpha _{1}^{n-1}\\1&\alpha _{2}&\alpha _{2}^{2}&\dots &\alpha _{2}^{n-1}\\1&\alpha _{3}&\alpha _{3}^{2}&\dots &\alpha _{3}^{n-1}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&\alpha _{m}&\alpha _{m}^{2}&\dots &\alpha _{m}^{n-1}\end{bmatrix}\end{align}

```julia
using PyCall
```

```julia
np = pyimport("numpy")
```

```julia
np.vander(1:5, increasing=true)
```

The source code for this function is [here](https://github.com/numpy/numpy/blob/v1.16.1/numpy/lib/twodim_base.py#L475-L563). It calls `np.multiply.accumulate` which is implemented in C [here](https://github.com/numpy/numpy/blob/deea4983aedfa96905bbaee64e3d1de84144303f/numpy/core/src/umath/ufunc_object.c#L3678). However, this code doesn't actually perform the computation, it basically only checks types and stuff. The actual kernel that gets called is [here](https://github.com/numpy/numpy/blob/deea4983aedfa96905bbaee64e3d1de84144303f/numpy/core/src/umath/loops.c.src#L1742). This isn't even C code but a template for C code which is used to generate type specific kernels.

Overall, this setup only supports a limited set of types, like `Float64`, `Float32`, and so forth.

Here is a simple Julia implementation

```julia
function vander(x::AbstractVector{T},  n=length(x)) where T
    m = length(x)
    V = Matrix{T}(undef, m, n)
    for j = 1:m
        V[j,1] = one(x[j])
    end
    for i= 2:n
        for j = 1:m
            V[j,i] = x[j] * V[j,i-1]
            end
        end
    return V
end
```

```julia
vander(1:5)
```

#### A quick speed comparison

  Show Code

```julia
using BenchmarkTools, Plots
ns = exp10.(range(1, 4, length=30));

tnp = Float64[]
tjl = Float64[]
for n in ns
    x = 1:n |> collect
    push!(tnp, @belapsed np.vander(\$x) samples=3 evals=1)
    push!(tjl, @belapsed vander(\$x) samples=3 evals=1)
end
plot(ns, tnp./tjl, m=:circle, xscale=:log10, xlab="matrix size", ylab="NumPy time / Julia time", legend=:false)
```

Note that the clean and concise Julia implementation is **beating numpy's C implementation for small matrices** and is **on-par for large matrix sizes**.

At the same time, the Julia code is *generic* and works for arbitrary types!

```julia
vander(Int32[4, 8, 16, 32])
```

It even works for non-numerical types. The only requirement is that the type has a *one* (identity element) and a multiplication operation defined.

```julia
vander(["this", "is", "a", "test"])
```

Here, `one(String) == ""` since the empty string is the identity under multiplication (string concatenation).

# How can Julia be fast?

**AST = Abstract Syntax Tree**

**SSA = Static Single Assignment**

**[LLVM](https://de.wikipedia.org/wiki/LLVM) = Low Level Virtual Machine**

### Specialization and code inspection

**Julia specializes on the types of function arguments.**

When a function is called for the first time, Julia compiles efficient machine code for the given input types.

If it is called again, the already existing machine code is reused, until we call the function with different input types.

```julia
func(x,y) = x^2 + y
```

```julia
@time func(1,2)
@time func(1,2)
```

**First call:** compilation + running the code

**Second call:** running the code

```julia
@time func(1,2)
```

If the input types change, Julia compiles a new specialization of the function.

```julia
@time func(1.3,4.8)
@time func(1.3,4.8)
```

We now have two efficient codes, one for all `Int64` inputs and another one for all `Float64` arguments, in the cache.

### *But I really want to see what happens!*

We can inspect the code at all transformation stages with a bunch of macros:

* The AST after parsing (**`@macroexpand`**)
* The AST after lowering (**`@code_typed`**, **`@code_warntype`**)
* The AST after type inference and optimization (**`@code_lowered`**)
* The LLVM IR (**`@code_llvm`**)
* The assembly machine code (**`@code_native`**)

```julia
@code_typed func(1,2)
```

```julia
@code_lowered func(1,2)
```

```julia
@code_llvm func(1,2)
```

We can remove the comments (lines starting with `;` using `debuginfo=:none`).

```julia
@code_llvm debuginfo=:none func(1,2)
```

```julia
@code_native debuginfo=:none func(1,2)
```

Let's compare this to `Float64` input.

```julia
@code_native debuginfo=:none func(1.2,2.9)
```

## How important is code specialization?

Let's try to estimate the performance gain by specialization.

We wrap our numbers into a custom type which internally stores them as `Any` to prevent specialization.

(This is qualitatively comparable to what Python does.)

```julia
struct Anything
    value::Any
end

operation(x::Number) = x^2 + sqrt(x)
operation(x::Anything) = x.value^2 + sqrt(x.value)
```

```julia
using BenchmarkTools

@btime operation(2);
@btime operation(2.0);

x = Anything(2.0)
@btime operation($x);
```

**That's about an 40 times slowdown!**

```julia
@code_native debuginfo=:none operation(2.0)
```

```julia
@code_native debuginfo=:none operation(x)
```

# Make run-time the fun time.

In scientific computations, we typically run a piece of code many times over and over again. Think of a Monte Carlo simulation, for example, where we perform the update and the Metropolis check millions of times.

**Therefore, we want our run-time to be as short as possible.**

On the other hand, for a given set of input arguments, Julia compiles the piece of code only once, as we have seen above. The time it takes to compile our code is almost always negligible compared to the duration of the full computation.

A general strategy is therefore to move parts of the computation to compile-time.

Since Julia specializes on types, at compile-time **only type information is available to the compiler.**

```julia
f1(x::Int) = x + 1
f2(x::Int) = x + 2

function f_slow(x::Int, p::Bool)
    if p                                # check depends on the value of p
        return f1(x)
    else
        return f2(x)
    end
end
```

```julia
@code_llvm debuginfo=:none f_slow(1, true)
```

We can eliminate the if branch by moving the condition check to the type domain. This way, it **will only be evaluated once at compile-time.**

```julia
abstract type Boolean end
struct True <: Boolean end # type domain true
struct False <: Boolean end # type domain false

function f_fast(x::Int, p::Boolean)
    if typeof(p) == True                # check solely based on the type of p
        return f1(x)
    else
        return f2(x)
    end
end
```

```julia
@code_llvm debuginfo=:none f_fast(1, True())
```

# Are explicit type annotations necessary? (like in C or Fortran)

Note that Julia's type inference is powerful. Specifying types **is not** necessary for best performance!

```julia
function my_function(x)
    y = rand()
    z = rand()
    x+y+z
end

function my_function_typed(x::Int)::Float64
    y::Float64 = rand()
    z::Float64 = rand()
    x+y+z
end
```

```julia
@btime my_function(10);
@btime my_function_typed(10);
```

 However, annotating types explicitly can serve a purpose.

* **Define a user interface/type filter** (will throw error if incompatible type is given)
* Enforce conversions
* Rarely, help the compiler infer types in tricky situations

# Core messages of this Notebook

* Julia **can be fast.**
* **A function is compiled when called for the first time** with a given set of argument types.
* The are **multiple compilation steps** all of which can be inspected through macros like `@code_warntype`.
* **Code specialization** based on the types of all of the input arguments is important for speed.
* Calculations can be moved to compile-time to make run-time faster.
* In virtually all cases, **explicit type annotations are irrelevant for performance**.
* Type annotations in function signatures define a **type filter/user interface**.


# Shor's Algorithm for factoring

In this tutorial, we will walk through Shor’s Algorithm using the quantum circuit model.
As we know already, the security of RSA is based on the use of a one-way function – prime factorization – which is related to the problem of period ﬁnding in modular exponentiation. As we saw, if one picks two prime numbers, $p$ and $q$, which multiply together to give a modulus $N$, then one can efﬁciently ﬁnd the modular exponentiation period $r$, and thus generate a public encryption key e and a private decryption key $d$. However, if we are just given the modulus $N$, then we are stuck, because there’s no known efﬁcient classical algorithm that lets us ﬁnd the prime factors $p$ and $q$, and so there’s no efﬁcient means to determine the period $r$. Thus, even if we reveal the modulus $N$ and the public encryption key $e$, there is no known efﬁcient way for an eavesdropper to calculate the private key $d$ on a classical computer. However, **if Eve has access to a quantum computer** that can run Shor’s Algorithm, then she can efﬁciently ﬁnd the modular exponentiation period r.

$$e\cdot d = 1 \mod r ~~~~~\rightarrow~~~~~ d = e^{−1} \mod r$$

This is called order ﬁnding, and Shor’s Algorithm ﬁnds the order $r$ of a number $a$ with respect to the modulus $N$,

$$a^x \mod N ~~~~~\rightarrow~~~~~ \text{order (period) } r,$$

where $a$ is the number that is raised to integer powers under modular exponentiation. With a means to efﬁciently determine $r$, Eve can calculate the prime numbers $p$ and $q$ and then derive the decryption key $d$, thus compromising RSA.

Let’s walk through Shor's algorithm, step by step, with the initialization stage. We need two registers of qubits for Shor’s Algorithm. Register 1 is used to store the results of a period ﬁnding protocol, which is implemented using the Quantum Fourier Transform. Register 2 is used to store the values of the modular exponentiation. This is the function that will apply in the compute section. The number of qubits in the ﬁrst register should generate a state space that is at least large enough to capture the maximum possible period, and do so at least twice to see it repeat. Let’s deﬁne N as twice the maximum period. So we need

$$2 L > 2r_{max} = N ~~~~~\rightarrow~~~~~ N^2 < 2^{2L} < 2N^2  ~~~~~\rightarrow~~~~~  2L \text{ qubits}.$$

The more qubits we have, the more densely we can sample the solution space, and thereby reduce the error in determining the period. Here, we’ll take 2L qubits for the ﬁrst register. And for simplicity, we’ll use the same number of qubits in the second register. By choosing 2L qubits, the number of states – $2^{2L} > N^2$. This guarantees that there are at least $N$ terms contributing to the probability amplitude when estimating the period, even as the period $r$ gets exponentially large, approaching $N/2$.

We prepare the qubits in state $|0\rangle^{\otimes 2L}$.

Next, at the compute stage, we use Hadamard gates to place the register 1 qubits into an equal superposition state. We again use the superscripted tensor product notation to indicate that we apply a Hadamard gate to each of the $2L$ qubits. This puts each qubit into a superposition state, $\frac{1}{\sqrt{2}}(| 0 \rangle + | 1\rangle)$. Multiplying out all $2L$ of these single qubit superposition states results in a single large equal superposition state with $2^{2L}$ components. From a component with $2L$ qubits in state $| 0 \rangle$ to a component with all qubits in state $| 1 \rangle$, and with all combinations in between. If we now number these components from $0$ to $2^{2L−1}$, we can rewrite the superposition state as a sum over $x$

$$
|0\rangle^{\otimes 2L}|0\rangle^{\otimes 2L} \nonumber \xrightarrow {\substack{H^{\otimes 2L}}}\frac{1}{\sqrt{2^{2L}}} \Big(|0\rangle + |1\rangle\Big)^{\otimes 2L} |0\rangle^{\otimes 2L}  =  \frac{1}{\sqrt{2^{2L}}}\Big(|00\cdots 0\rangle + \cdots + |11\cdots 1\rangle\Big)|0\rangle^{\otimes 2L}   = \frac{1}{\sqrt{2^{2L}}}\sum_{x=0}^{2^{2L}-1}|x\rangle|0\rangle^{\otimes 2L}.
$$

In fact, each decimal number $x$ corresponds to one of the binary numbers represented by the $2L$ qubits. Next, the function $f(x) = a^x \mod N$ is performed:

$$
\frac{1}{\sqrt{2^{2L}}}\sum_{x=0}^{2^{2L}-1}|x\rangle|0\rangle^{\otimes 2L} \xrightarrow {\substack{U_{f(x)}}} \frac{1}{\sqrt{2^{2L}}}\sum_{x=0}^{2^{2L}-1}|x\rangle|f(x)\rangle =  \frac{1}{\sqrt{2^{2L}}}\sum_{x=0}^{2^{2L}-1}|x\rangle|a^x \mod N\rangle.
$$

$f(x)$ implements modular exponentiation, which outputs a number $a^x \mod N$. We will present in detail how the modular exponentiation is implemented in a real circuit below. In fact, there are multiple approaches to doing it based on reversible computing and multiplication algorithms. The approach we adopt is based on [this article](https://arxiv.org/pdf/quant-ph/0205095.pdf). We note that this function is implemented using ancilla qubits and it’s efﬁcient, requiring a number of gates that scales only polynomially with the number of qubits.
The number $a$ is chosen at random, and it must be co-prime with $N$. That is, it must have no common factors with $N$. The resulting output values from the function $f$ are then stored in the Register 2 qubits. Since the values of the power $x$ are taken from the qubits in the ﬁrst register, implementing $f(x)$ makes a conditional connection between the qubits in Register 1 and those in Register 2. Although we ultimately won’t need to measure the qubits in Register 2, this step sets the stage for quantum interference and quantum parallelism to occur in Register 1 at the next step, which is the Quantum Fourier Transform. Here is how the circuit looks like. Below we will implement each building block of the circuit.

## Implementation of the Modular Exponentiation

We will adopt the approach presented in [this article](https://arxiv.org/pdf/quant-ph/0205095.pdf) which uses Kitaev's version of order finding in Shor's algorithm, **Phase Estimation Algorithm** as a subroutine.

The state $|u\rangle$ must be the eigenstate or a superposition of eigenstate of the unitary $U$ for the algorithm to work properly. As stated, our unitary does the following transformation on the bottom register (multiplies by $a$ to the content of the register, mod N):

$$
U|y\rangle = |ay \mod N\rangle.
$$

It can be shown that the eigenvalue has the following form (you can see it by checking $U|u\rangle = \lambda |u\rangle$)

$$
|u_s\rangle = \frac{1}{\sqrt{r}} \sum_{k=0}^{r-1}\exp{\Big[ \frac{-2\pi i sk}{r}\Big]}|x^k \mod N\rangle, ~~~ s=0,1,\cdots r-1.
$$

Preparing this state is not an easy task experimentally. We note that

$$ \frac{1}{\sqrt{r}} \sum_{k=0}^{r-1}|u_s\rangle = |1\rangle,$$

which ($|1\rangle$) is a superposition of eigenstates and pretty easy to construct. Here is the required circuit.

To implement the unitary $U$ that carries out transformation $|x\rangle|0\rangle \rightarrow |x\rangle|ax \text{ mod } N\rangle$, which uses the `controlled_mult_mod_N`, the article implements the following cicuit

that uses another circuit, an adder in Fourier space (the function to be implemented below `add_in_fourier_space` as well as `controlled_controlled_add_in_fourier_space`)

The addition in Fourier domain looks like the following

And if needed we can transform back by conjugating with quantum fourier transform (defined below)

### Implement the above circuits from bottom to top

```python
from qiskit import QuantumCircuit, ClassicalRegister, QuantumRegister
from qiskit import execute, IBMQ
from qiskit import BasicAer

import sys

import math
import random
import array
import fractions
import numpy as np
```

### Criate Circuit for Addition in Fourier domain

```python
# first define a helper function to use below
def get_angles(a, N):
    """Function that calculates the array of angles
    to be used in the addition in Fourier Space"""

    s = bin(int(a))[2:].zfill(N)
    angles = np.zeros([N])
    for i in range(0, N):
        for j in range(i, N):
            if s[j] == '1':
                angles[N-i-1] += math.pow(2, -(j-i))
        angles[N-i-1] *= np.pi

    return angles

```

```python
def add_in_fourier_space(circuit, q, a, N, inv = False):
    """Creates the circuit that performs addition a in Fourier Space
    Can also be used for subtraction by setting
    the parameter inv to True"""

    angle = get_angles(a, N)
    for i in range(0, N):
        if inv == False:
            circuit.u1(angle[i], q[i])
        else:
            circuit.u1(-angle[i], q[i])
```

### Create Circuit for Controlled Controlled Addition in Fourier space

In order to implement the `Controlled Controlled Addition mod N`, we see that we need QFT, inverse QFT, `Controlled Addition` as well as `Controlled Controlled Addition`. Let's first implement the `Controlled Addition` and `Controlled Controlled Addition` gates and then cover the QFT theory and implementation of `Controlled Controlled Addition mod N`.

```python
def controlled_add_in_fourier_space(circuit, q, ctrl, a, n, inv=False):
    """Single controlled version of the add_in_fourier_space() circuit"""

    angle = get_angles(a, n)
    for i in range(n):
        if inv == False:
            circuit.cu1(angle[i], ctrl, q[i])
        else:
            circuit.cu1(-angle[i], ctrl, q[i])

```

```python
def ccphase(circuit, angle, ctrl1, ctrl2, tgt):
    """Creates a doubly controlled phase gate,
    a helper function for doubly controlled addition"""

    circuit.cu1(angle/2, ctrl1, tgt)
    circuit.cx(ctrl2, ctrl1)
    circuit.cu1(-angle/2, ctrl1, tgt)
    circuit.cx(ctrl2, ctrl1)
    circuit.cu1(angle/2, ctrl2, tgt)
```

```python
def controlled_controlled_add_in_fourier_space(circuit, q, ctrl1, ctrl2, a, n, inv=False):
    """Doubly controlled version of the add_in_fourier_space() circuit"""

    angle = get_angles(a, n)
    for i in range(n):
        if inv == False:
            ccphase(circuit, angle[i], ctrl1, ctrl2, q[i])
        else:
            ccphase(circuit, -angle[i], ctrl1, ctrl2, q[i])
```

Now let's get intuition on QFT and implement it. Then we will combine it with the above gate to implement the `Controlled Controlled Addition` circuit.

## Quantum Fourier Transform

The Quantum Fourier Transform is a quantum version of the classical [discrete time Fourier Transform](https://en.wikipedia.org/wiki/Discrete-time_Fourier_transform). The classical Fourier Transform takes a vector of numbers, $x$ -- often a [digital sampling of a signal](https://en.wikipedia.org/wiki/Sampling_(signal_processing)) to be analyzed -- and transforms it to a vector of numbers, $y$, in the frequency domain.

\begin{equation}
y_k \equiv \frac{1}{\sqrt{N}}\sum_{j=0}^{N-1}x_je^{i2\pi jk/N}
\end{equation}

Periodic behavior in the time domain is more easily identified in the transformed [frequency domain](https://en.wikipedia.org/wiki/Frequency_domain), which is why we do this. For example, a sine wave that oscillates in time with a specific frequency will transform to a large amplitude spike at plus or minus that frequency in the transform domain. Thus, it's easy to identify the location of the spikes and then read off the frequency directly. And if we know a signal's frequency, then we also know its period, since the frequency is $1/\text{period}$.\\
Now, the Quantum Fourier Transform is essentially the same transformation, which is why we'll use it here for period finding. It takes a state $|j\rangle$ and transforms it to a superposition state with specific phase factors.

\begin{equation}
|j\rangle \rightarrow \frac{1}{\sqrt{N}}\sum_{k=0}^{N-1}e^{i2\pi jk/N}|k\rangle
\end{equation}

If we start with a superposition state with coefficients $x_j$, the transformed result is also a superposition state with coefficients $y_k$, where the \textit{probability amplitudes $y_k$ are the classical discrete time Fourier Transform of the probability amplitudes $x_j$}.

\begin{equation}
\sum_{j=0}^{N-1}x_j|j\rangle \rightarrow \sum_{k=0}^{N-1}\frac{1}{\sqrt{N}}\sum_{j=0}^{N-1}x_je^{i2\pi jk/N}|k\rangle = \sum_{k=0}^{N-1}y_k|k\rangle
\end{equation}

And due to quantum parallelism and quantum interference, implementing the Quantum Fourier Transform is exponentially faster than implementing the classical fast Fourier Transform algorithm. That sounds great, but there's an important catch related to quantum measurement.
As you know, the measurement of a quantum state is probabilistic. Although the output of the Quantum Fourier Transform is a large superposition state with transformed probability amplitudes, we generally can't access those amplitudes. Our measurement projects out only a single state, and even if we identically prepare the system and measure it many times, we only learn about the magnitude squared of the corresponding coefficients. We lose the phase information. So we don't have access to the probability amplitudes in the quantum computer, and this is quite different from the classical Fourier Transform where the entire output vector of complex numbers can be directly read. Thus, the Quantum Fourier Transform is most useful in problems where there's a unique period or phase that can be found with high probability.

If we represent the ket $|j\rangle$ in its binary representation $|j_1\rangle\otimes |j_2\rangle\otimes\cdots \otimes |j_n\rangle$, then the $n$-qubit output state can be written as the product ofnsingle-qubit states (for the derivation, see chapter 5 of Nilsen $\&$ Chuang, Eqs (5.5)-(5.10))

\begin{eqnarray}\label{eqn2}
\left\vert j_{1}\right\rangle \otimes \left\vert j_{2}\right\rangle \otimes &&\nonumber...\otimes \left\vert  j_{n-1}\right\rangle  \otimes \left\vert j_{n}\right\rangle \rightarrow
\left( \frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{n}\right) \left\vert 1\right\rangle }{\sqrt{2}}\right) \otimes \left( \frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{n-1}j_{n}\right) \left\vert 1\right\rangle }{\sqrt{2}}\right) \\&& \otimes ...\otimes \left( \frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{2}...j_{n}\right) \left\vert 1\right\rangle }{\sqrt{2}}\right) \otimes \left( \frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{1}...j_{n}\right) \left\vert 1\right\rangle }{\sqrt{2}}\right).
\end{eqnarray}

The figure below shows a graphical representation of the Quantum Fourier Transform for an $n$-qubit system.

The figure below shows the circuit implementation of the Quantum Fourier Transform in terms of multiple Hadamard gates and controlled gates.

The phase rotation $R_k$ is defined as

$$R_k\lvert 0\rangle = \lvert 0\rangle,$$

$$R_k\lvert 1\rangle = \displaystyle \exp \left(i\frac{2\pi }{2^{k}}\right) \left\vert 1\right\rangle .$$

To return the qubit states to their original ordering, we need to apply **SWAP**-gates, which aregenerally used to exchange qubit states.

Let's analyze how the system evolves under this implementation.

A Hadamard gate is applied on the first qubit

$$\lvert j_1 \rangle \rightarrow \frac{\lvert 0 \rangle + \exp (2\pi i0.j_1)\lvert 1 \rangle }{\sqrt{2}},$$

and this puts the system in state

$$\frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{1}\right) \left\vert 1\right\rangle }{\sqrt {2}}\otimes \left\vert j_{2}\right\rangle \otimes \left\vert j_{3}\right\rangle \otimes ...\otimes \left\vert j_{n}\right\rangle .$$

 A controlled phase  $R_2$-gate is applied with the second qubit as control and the first as target,

$$\frac{\left\vert 0 \right \rangle + \exp \left(2\pi i0.j_1 j_2\right) \left\lvert 1 \right\rangle }{\sqrt{2}}\otimes \left\lvert j_2 \right\rangle \otimes \left\lvert j_3 \right \rangle \otimes ... \otimes \left\vert j_n \right\rangle .$$

A controlled phase $R_3$-gate is applied with the third qubit as control and the first as target,

$$\frac{\left\vert 0\right\rangle + \exp \left(2\pi i0.j_1 j_2 j_3 \right) \left \vert 1 \right \rangle}{\sqrt{2}} \otimes \left\vert j_2 \right \rangle \otimes \left\vert j_3 \right\rangle \otimes ... \otimes \left\vert j_n \right\rangle.$$

This continues until a controlled phase  $R_n$-gate is applied with the last qubit as control and the first as target, leaving the system in the state

$$\frac{\left\vert 0 \right \rangle + \exp \left(2\pi i0.j_1 j_2 j_3 ... j_n\right) \left\vert 1 \right\rangle}{\sqrt{2}}\otimes \left\vert j_2 \right\rangle \otimes \left\vert j_3 \right\rangle \otimes ... \otimes \left\vert j_n \right\rangle.$$

A Hadamard gate is applied on the second qubit,

$$\left (\frac{\left\vert0\right\rangle+\exp \left (2 \pi i0.j_1j_2j_3...j_n\right) \left\vert 1 \right\rangle}{\sqrt{2}}\right)\otimes \left(\frac{\left\vert 0 \right\rangle + \exp \left(2 \pi i0.j_2\right) \left\vert 1 \right\rangle}{\sqrt{2}}\right) \otimes \left\vert j_3 \right\rangle \otimes ... \otimes \left\vert j_n\right\rangle.$$

A controlled phase  $R_2$ -gate is applied with the third qubit as control and the second as target,

$$\left(\frac{\left\vert 0 \right\rangle + \exp \left (2 \pi i0.j_1 j_2 j_3 ... j_n\right) \left\vert 1 \right\rangle}{\sqrt{2}}\right) \otimes \left(\frac{\left\vert 0 \right\rangle + \exp \left(2\pi i0.j_2 j_3 \right) \left\vert 1 \right\rangle}{\sqrt{2}}\right) \otimes \left\vert j_3\right\rangle\otimes ... \otimes \left\vert j_n \right\rangle.$$

This continues until a controlled phase  $R_{n-1}$-gate is applied with the last qubit as control and the second as target, leaving the system in the state

$$\frac{\left\vert 0 \right\rangle + \exp \left(2\pi i0.j_1 j_2 j_3 ... j_n \right) \left\vert 1 \right\rangle}{\sqrt{2}}\otimes\frac{\left\vert 0\right\rangle + \exp \left(2 \pi i0.j_2 j_3 ... j_n\right) \left\vert 1 \right\rangle }{\sqrt{2}} \otimes \left\vert j_3 \right\rangle \otimes ... \otimes \left\vert j_n\right\rangle.$$

This same process is done on each qubit until the last qubit, on which you can only apply a Hadamard gate. This implementation transforms as
\begin{eqnarray}\label{eqn3}
\left\vert j_{1}\right\rangle \otimes \left\vert j_{2}\right\rangle \otimes ...\otimes \left\vert j_{n-1}\right\rangle \otimes \left\vert j_{n}\right\rangle &&\rightarrow \left( \frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{1}...j_{n}\right) \left\vert 1\right\rangle }{\sqrt{2}}\right) \\&&\nonumber\otimes \left( \frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{2}...j_{n}\right) \left\vert 1\right\rangle }{\sqrt{2}}\right)
\otimes ...\\&&\nonumber \otimes \left( \frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{n-1}j_{n}\right) \left\vert 1\right\rangle }{\sqrt{2}}\right) \\&&\nonumber
\otimes \left( \frac{\left\vert 0\right\rangle +\exp \left( 2\pi i0.j_{n}\right) \left\vert 1\right\rangle }{\sqrt{2}}\right) .
\end{eqnarray}

To returnthe qubit states to their original ordering, we need to apply swap-gates, which are generally usedto exchange qubit states.

### Implement QFT and Inverse QFT

```python
def create_QFT(circuit, reg, n , with_swaps = False):
    """ Function to create QFT """

    # Apply the H gates and Cphases
    """ The Cphases with |angle| < threshold are not created because they do
    nothing. The threshold is put as being 0 so all CPhases are created,
    but the clause is there so if wanted just need to change the 0 of the
    if-clause to the desired value """
    i = n - 1
    while i >= 0:
        circuit.h(up_reg[i])
        j = i - 1
        while j >= 0:
            if (np.pi) / (pow(2, (i-j))) > 0:
                circuit.cu1((np.pi)/(pow(2, (i-j))), reg[i], reg[j]) # cu1(angle, ctrl, trgt)
                j = j - 1
        i = i - 1

    # If specified, apply the Swaps at the end
    if with_swaps == True:
        i=0
        while i < ((n-1) / 2):
            circuit.swap(reg[i], reg[n-1-i])
            i = i + 1

```

```python
def create_inverse_QFT(circuit, reg, n, with_swaps=False):
    """ Function to create inverse QFT """

    # If specified, apply the Swaps at the beggining
    if with_swaps == True:
        i = 0
        while i < ((n-1)/2):
            circuit.swap(reg[i], reg[n-1-i])
            i = i + 1

    """ Apply the H gates and Cphases"""
    i=0
    while i < n:
        circuit.h(reg[i])
        if i != n-1:
            j = i + 1
            y = i
            while y>=0:
                 if (np.pi) / (pow(2, (j-y))) > 0:
                    circuit.cu1(-(np.pi) / (pow(2, (j-y))), reg[j], reg[y])
                    y = y - 1
        i = i + 1
```

### Going back to implementing the CC adder and the "mod N" version of it

Let's implement the circuit and it's inverse.

```python
def controlled_controlled_add_mod_N(circuit, q, ctrl1, ctrl2, aux, a, N, n):
    """Circuit that implements doubly controlled modular addition by a
    args: self explanatory from the picture"""

    controlled_controlled_add_in_fourier_space(circuit, q, ctrl1, ctrl2, a, n)
    add_in_fourier_space(circuit, q, N, n, inv=True)
    create_inverse_QFT(circuit, q, n)
    circuit.cx(q[n-1], aux)
    create_QFT(circuit, q, n)
    controlled_add_in_fourier_space(circuit, q, aux, N, n)

    controlled_controlled_add_in_fourier_space(circuit, q, ctrl1, ctrl2, a, n, inv=True)
    create_inverse_QFT(circuit, q, n)
    circuit.x(q[n-1])
    circuit.cx(q[n-1], aux)
    circuit.x(q[n-1])
    create_QFT(circuit, q, n)
    controlled_controlled_add_in_fourier_space(circuit, q, ctrl1, ctrl2, a, n)

def controlled_controlled_add_mod_N_inv(circuit, q, ctrl1, ctrl2, aux, a, N, n):
    """Circuit that implements the inverse of doubly controlled modular addition by a"""

    controlled_controlled_add_in_fourier_space(circuit, q, ctrl1, ctrl2, a, n, inv=True)
    create_inverse_QFT(circuit, q, n)
    circuit.x(q[n-1])
    circuit.cx(q[n-1], aux)
    circuit.x(q[n-1])
    create_QFT(circuit, q, n)
    controlled_controlled_add_in_fourier_space(circuit, q, ctrl1, ctrl2, a, n)

    controlled_add_in_fourier_space(circuit, q, aux, N, n, inv=True)
    create_inverse_QFT(circuit, q, n)
    circuit.cx(q[n-1], aux)
    create_QFT(circuit, q, n)
    add_in_fourier_space(circuit, q, N, n)
    controlled_controlled_add_in_fourier_space(circuit, q, ctrl1, ctrl2, a, n, inv=True)
```

### Implement the controlled multiplication mod N circuit

```python
# helper functions
def euclid_gcd(a, b):
    if a == 0:
        return (b, 0, 1)
    else:
        g, y, x = euclid_gcd(b%a, a)
        return (g, x-(b//a)*y, y)
def mod_inv(a, m):
    g, x, y = euclid_gcd(a, m)
    if g != 1:
        raise Exception('modular inverse does not exist')
    else:
        return x % m

def controlled_mult_mod_N(circuit, ctrl, q, aux, a, N, n):
    """Circuit that implements single controlled modular multiplication by a
    args: self explanatory from the picture"""

    # CMULT(a) mod N
    create_QFT(circuit, aux, n+1)
    for i in range(0, n):
        controlled_controlled_add_mod_N(circuit, aux, q[i], ctrl, aux[n+1], (2**i)*a % N, N, n+1)
    create_inverse_QFT(circuit, aux, n+1, 0)

    for i in range(0, n):
        circuit.cswap(ctrl, q[i], aux[i])

    # CMULT(a^{-1}) mod N,
    # this is done to make sure circuit is reversible so that we can use the qubits further
    a_inv = mod_inv(a, N)
    create_QFT(circuit, aux, n+1, 0)
    i = n-1
    while i >= 0:
        controlled_controlled_add_mod_N_inv(circuit, aux, q[i], ctrl, aux[n+1], math.pow(2,i)*a_inv % N, N, n+1)
        i -= 1
    create_inverse_QFT(circuit, aux, n+1, 0)

```

### Going back to the original Shor's circuit and measurement of the Register 1

Now, returning to Shor's Algorithm, the Quantum Fourier Transform is applied here to the qubits in Register 1, and it imparts specific phase factors that are related to both $x$ and $z$.

\begin{eqnarray}
  \frac{1}{\sqrt{2^{2L}}}\sum_{x=0}^{2^{2L-1}}|x\rangle|a^x \mod N\rangle \xrightarrow {\substack{QFT}} &&\nonumber     \frac{1}{\sqrt{2^{2L}}} \sum_{z=0}^{2^{2L-1}}\sum_{x=0}^{2^{2L-1}} e^{i2\pi xz/2^{2L}}|z\rangle|a^x \mod N\rangle \\&&\nonumber =  \frac{1}{\sqrt{2^{2L}}}\sum_{z=0}^{2^{2L-1}}\sum_{x=0}^{2^{2L-1}} \omega^{xz}|z\rangle|a^x \mod N\rangle
\end{eqnarray}

That is, they're related to both the coefficients of the qubits on Register 1 and the modular exponentiation stored on Register 2. The phase sampling increment is represented here by $\omega$, and it's the phase

$$\frac{2\pi}{2^{2L}}.$$

Here we can see that increasing the number of qubits ($L$) will reduce the step size, and thus the error in estimating the actual period. Essentially, more qubits will more densely sample the phases that represent the period.
Lastly, we measure the qubits in Register 1. The projected state that results is a binary value of $z$, and its decimal value is

$$z_{\text{decimal}} = \text{integer} ~ \times 2^{2L}/r. $$

Thus, it can be used to find the period $r$ using a [continued fraction expansion](https://en.wikipedia.org/wiki/Continued_fraction) of $z/2^{2L}$.

## Putting everyting together

## Postprocessing the measurement and finding factors p and q of N=pq

##### Here are the steps of Shor's algorithm

1. randomly pick a number, $a$, that's smaller than $N$ and relatively prime. That is, their greatest common divisor--$\gcd (a,N)=1$.

2. We then use Shor's Algorithm to find the period of $a^x \mod N$. Shor's Algorithm, with classical post-processing, will efficiently give us the period $r$.
    Before proceeding, we need to do a couple checks.
    $r$ must be an even number
    we need to ensure that $a^{r/2} + 1 \neq 0 \mod N$.
    If we fail either of these tests, we need to go back to step 1 and pick another value of $a$.

3. We know that $$a^r = 1 \mod N,$$ as this is a property of modular exponentiation. We can rewrite this expression as $$a^r - 1 = k N$$ for an integer $k$.

4. Replace $N$ with the unknown prime factors $p$ and $q$, and we factor the left-hand side to yield

    $$a^r-1 = kpq.$$
    $$(a^{r/2}+1)(a^{r/2}-1) = kpq$$

5. Finally solve for $p$ and $q$:

    $$p=\gcd (a^{r/2}+1, N)$$
    $$q=\gcd (a^{r/2}-1, N)$$

```python
# Choose N
N = 15

def get_value_a(N):
    """ Function to get the value a (1 < a < N), such that a and N are coprime. """
    # YOUR CODE HERE

# choose a
a = get_value_a(N)

# define number of qubits
n = math.ceil(math.log(N, 2))
print('Total number of qubits used: {0}\n'.format(4*n + 2))
```

```python
# Let's create quantum and classical registers

# auxilliary quantum register used in addition and multiplication
aux = QuantumRegister(n + 2)

# quantum register where the sequential QFT is performed
up_reg = QuantumRegister(2 * n)

# quantum register where the multiplications are made
down_reg = QuantumRegister(n)

# classical register where the measured values of the QFT are stored
classic_reg = ClassicalRegister(2 * n)

# Create Quantum Circuit!
circuit = QuantumCircuit(down_reg , up_reg , aux, classic_reg)

# Initialize down register to |1> (see pic)
# and create maximal superposition in top register
circuit.h(up_reg)
circuit.x(down_reg[0])

# Apply the multiplication gates in order to create the exponentiation
for i in range(0, 2*n):
    controlled_mult_mod_N(circuit, up_reg[i], down_reg, aux, int(pow(a, pow(2, i))), N, n)

# Apply inverse QFT
create_inverse_QFT(circuit, up_reg, 2*n ,1)

# Measure the top qubits, to get value C
circuit.measure(up_reg, classic_reg)
```

```python
def continued_fractions(C, K):
    """
    implements the cont frac algo
    C = a_0 + 1/(a_1 + 1/(a_2 + 1/(a_3 + .../(a_{K-1} + 1/a_K))))
    returns the list of [a_0, a_1, ..., a_K]
    """
    a = []
    # YOUR CODE HERE
    return a

def convergents(a):
    """
    Finds convergents from coefficients
    param a: list, output of function continued_fractions(C, K)
    """
    print('a=: ', a)
    z = []
    r = []

    # set the first two elements: z0 = a0, z1=a1a0 + 1, r0=1, r1=a1
    z.append(a[0])
    r.append(1)
    z.append(a[1] * a[0] + 1)
    r.append(a[1])

    # YOUR CODE HERE

    return z, r

def check_and_return_period(K=10):
    """ checks if f(x)=a^{r_k} mod N = 1
    returns r period"""
    # YOUR CODE HERE

def find_factors(r):
    # YOUR CODE HERE

```

### Run everyting and find factors

```python
""" Simulate the created Quantum Circuit """
# to run on IBM, use backend=IBMQ.get_backend('ibmq_qasm_simulator') in execute() function
# to run locally, use backend=BasicAer.get_backend('qasm_simulator') in execute() function
simulation = execute(circuit, backend=BasicAer.get_backend('qasm_simulator'), shots=number_shots)

# Get the results of the simulation in proper structure
sim_result=simulation.result()
counts_result = sim_result.get_counts(circuit)

# Get the C value from the final state qubits
output_desired = list(sim_result.get_counts().keys())[i]
print('output_desired=: ', output_desired)

C  = int(output_desired, 2)
print('C=: ', C)

result = (int(list(sim_result.get_counts().values())[i] ) )

print("------> Analysing result {0}. \
This result happened in {1:.4f} % of all cases\n".format(output_desired, result))

# Print the final C value to user
print('In decimal, C value for this result is: {0}\n'.format(C))

# Get the period using the value obtained
period = check_and_return_period(C, 10, N)
print('period is : ', period)

# Get the factors using the period r
factors = find_factors(period)
print('the factors are : ', factors)
```

Shor/.ipynb_checkpoints/Shor's Algorithm for factoring-checkpoint.ipynb


# Inference with GPs

The dataset needed for this worksheet [can be downloaded](https://northwestern.box.com/s/el0s1imhdxq5qwvzb4hgap90mxpjcfdq). Once you have downloaded [s9_gp_dat.tar.gz](https://northwestern.box.com/s/el0s1imhdxq5qwvzb4hgap90mxpjcfdq), and moved it to this folder, execute the following cell:

```python
!tar -zxvf s9_gp_dat.tar.gz
!mv *.txt data/
```

Here are the functions we wrote in the previous tutorial to compute and draw from a GP:

```python
import numpy as np
from scipy.linalg import cho_factor

def ExpSquaredKernel(t1, t2=None, A=1.0, l=1.0):
    """
    Return the ``N x M`` exponential squared
    covariance matrix between time vectors `t1`
    and `t2`. The kernel has amplitude `A` and
    lengthscale `l`.

    """
    if t2 is None:
        t2 = t1
    T2, T1 = np.meshgrid(t2, t1)
    return A ** 2 * np.exp(-0.5 * (T1 - T2) ** 2 / l ** 2)

def draw_from_gaussian(mu, S, ndraws=1, eps=1e-12):
    """
    Generate samples from a multivariate gaussian
    specified by covariance ``S`` and mean ``mu``.

    (We derived these equations in Day 1, Notebook 01, Exercise 7.)
    """
    npts = S.shape[0]
    L, _ = cho_factor(S + eps * np.eye(npts), lower=True)
    L = np.tril(L)
    u = np.random.randn(npts, ndraws)
    x = np.dot(L, u) + mu[:, None]
    return x.T

def compute_gp(t_train, y_train, t_test, sigma=0, A=1.0, l=1.0):
    """
    Compute the mean vector and covariance matrix of a GP
    at times `t_test` given training points `y_train(t_train)`.
    The training points have uncertainty `sigma` and the
    kernel is assumed to be an Exponential Squared Kernel
    with amplitude `A` and lengthscale `l`.

    """
    # Compute the required matrices
    kernel = ExpSquaredKernel
    Stt = kernel(t_train, A=1.0, l=1.0)
    Stt += sigma ** 2 * np.eye(Stt.shape[0])
    Spp = kernel(t_test, A=1.0, l=1.0)
    Spt = kernel(t_test, t_train, A=1.0, l=1.0)

    # Compute the mean and covariance of the GP
    mu = np.dot(Spt, np.linalg.solve(Stt, y_train))
    S = Spp - np.dot(Spt, np.linalg.solve(Stt, Spt.T))

    return mu, S
```

## The Marginal Likelihood

In the previous notebook, we learned how to construct and sample from a simple GP. This is useful for making predictions, i.e., interpolating or extrapolating based on the data you measured. But the true power of GPs comes from their application to *regression* and *inference*: given a dataset $D$ and a model $M(\theta)$, what are the values of the model parameters $\theta$ that are consistent with $D$? The parameters $\theta$ can be the hyperparameters of the GP (the amplitude and time scale), the parameters of some parametric model, or all of the above.

A very common use of GPs is to model things you don't have an explicit physical model for, so quite often they are used to model "nuisances" in the dataset. But just because you don't care about these nuisances doesn't mean they don't affect your inference: in fact, unmodelled correlated noise can often lead to strong biases in the parameter values you infer. In this notebook, we'll learn how to compute likelihoods of Gaussian Processes so that we can *marginalize* over the nuisance parameters (given suitable priors) and obtain unbiased estimates for the physical parameters we care about.

Given a set of measurements $y$ distributed according to
$$
\begin{align}
    y \sim \mathcal{N}(\mathbf{\mu}(\theta), \mathbf{\Sigma}(\alpha))
\end{align}
$$
where $\theta$ are the parameters of the mean model $\mu$ and $\alpha$ are the hyperparameters of the covariance model $\mathbf{\Sigma}$, the *marginal likelihood* of $y$ is
$$
\begin{align}
    \ln P(y | \theta, \alpha) = -\frac{1}{2}(y-\mu)^\top \mathbf{\Sigma}^{-1} (y-\mu) - \frac{1}{2}\ln |\mathbf{\Sigma}| - \frac{N}{2} \ln 2\pi
\end{align}
$$

where $||$ denotes the determinant and $N$ is the number of measurements. The term *marginal* refers to the fact that this expression implicitly integrates over all possible values of the Gaussian Process; this is not the likelihood of the data given one particular draw from the GP, but given the ensemble of all possible draws from $\mathbf{\Sigma}$.

Exercise 1

Define a function ``ln_gp_likelihood(t, y, sigma, A=1, l=1)`` that returns the log-likelihood defined above for a vector of measurements ``y`` at a set of times ``t`` with uncertainty ``sigma``. As before, ``A`` and ``l`` should get passed direcetly to the kernel function. Note that you're going to want to use [np.linalg.slogdet](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.linalg.slogdet.html) to compute the log-determinant of the covariance instead of ``np.log(np.linalg.det)``. (Why?)

```python
def ln_gp_likelihood(t, y, sigma=0, A=1.0, l=1.0):
    """

    """
    # do stuff in here
    pass
```

Exercise 2

The following dataset was generated from a zero-mean Gaussian Process with a Squared Exponential Kernel of unity amplitude and unknown timescale. Compute the marginal log likelihood of the data over a range of reasonable values of $l$ and find the maximum. Plot the **likelihood** (not log likelihood) versus $l$; it should be pretty Gaussian. How well are you able to constrain the timescale of the GP?

```python
import matplotlib.pyplot as plt
t, y, sigma = np.loadtxt("data/sample_data.txt", unpack=True)
plt.plot(t, y, "k.", alpha=0.5, ms=3)
plt.xlabel("time")
plt.ylabel("data");
```

Exercise 3a

The timeseries below was generated by a linear function of time, $y(t)= mt + b$. In addition to observational uncertainty $\sigma$ (white noise), there is a fair bit of correlated (red) noise, which we will assume is well described
by the squared exponential covariance with a certain (unknown) amplitude $A$ and timescale $l$.

Your task is to estimate the values of $m$ and $b$, the slope and intercept of the line, respectively. In this part of the exercise, **assume there is no correlated noise.** Your model for the $n^\mathrm{th}$ datapoint is thus

$$
\begin{align}
    y_n \sim \mathcal{N}(m t_n + b, \sigma_n\mathbf{I})
\end{align}
$$

and the probability of the data given the model can be computed by calling your GP likelihood function:

```python
def lnprob(params):
    m, b = params
    model = m * t + b
    return ln_gp_likelihood(t, y - model, sigma, A=0, l=1)
```

Note, importantly, that we are passing the **residual vector**, $y - (mt + b)$, to the GP, since above we coded up a zero-mean Gaussian process. We are therefore using the GP to model the **residuals** of the data after applying our physical model (the equation of the line).

To estimate the values of $m$ and $b$ we could generate a fine grid in those two parameters and compute the likelihood at every point. But since we'll soon be fitting for four parameters (in the next part), we might as well upgrade our inference scheme and use the ``emcee`` package to do Markov Chain Monte Carlo (MCMC). If you haven't used ``emcee`` before, check out the first few tutorials on the [documentation page](https://emcee.readthedocs.io/en/latest/). The basic setup for the problem is this:

```python
import emcee
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob)

initial = [4.0, 15.0]
p0 = initial + 1e-3 * np.random.randn(nwalkers, ndim)

print("Running burn-in...")
p0, _, _ = sampler.run_mcmc(p0, nburn)   # nburn = 500 should do
sampler.reset()

print("Running production...")
sampler.run_mcmc(p0, nsteps);            # nsteps = 1000 should do
```

where ``nwalkers`` is the number of walkers (something like 20 or 30 is fine), ``ndim`` is the number of dimensions (2 in this case), and ``lnprob`` is the log-probability function for the data given the model. Finally, ``p0`` is a list of starting positions for each of the walkers. Above we picked some fiducial/eyeballed value for $m$ and $b$, then added a small random number to each to generate different initial positions for each walker. This will initialize all walkers in a ball centered on some point, and as the chain progresses they'll diffuse out and begin to explore the posterior.

Once you have sampled the posterior, plot several draws from it on top of the data. You can access a random draw from the posterior by doing

```python
m, b = sampler.flatchain[np.random.randint(len(sampler.flatchain))]
```

Also plot the **true** line that generated the dataset (given by the variables ``m_true`` and ``b_true`` below). Do they agree, or is there bias in your inferred values? Use the ``corner`` package to plot the joint posterior. How many standard deviations away from the truth are your inferred values?

```python
t, y, sigma = np.loadtxt("data/sample_data_line.txt", unpack=True)
m_true, b_true, A_true, l_true = np.loadtxt("data/sample_data_line_truths.txt", unpack=True)
plt.errorbar(t, y, yerr=sigma, fmt="k.", label="observed")
plt.plot(t, m_true * t + b_true, color="C0", label="truth")
plt.legend(fontsize=12)
plt.xlabel("time")
plt.ylabel("data");
```

Exercise 3b

This time, let's actually model the correlated noise. Re-define your ``lnprob`` function to accept four parameters (slope, intercept, amplitude, and timescale). If you didn't before, it's a good idea to enforce some priors to keep the parameters within reasonable (and physical) ranges. If any parameter falls outside this range, have ``lnprob`` return negative infinity (i.e., zero probability).

You'll probably want to run your chains for a bit longer this time, too. As before, plot some posterior samples for the line, as well as the corner plot. How did you do this time? Is there any bias in your inferred values? How does the variance compare to the previous estimate?

Exercise 3c

If you didn't do this already, re-plot the posterior samples on top of the data, but this time draw them from the GP, *conditioned on the data*. How good is the fit?


# Project 2
## least squares regression and nearest neighbor classifiers

### Members:
* Akhilesh Vyas
* Pooja Bhatia
* Asif Altaf
* Abakar Bouba
* Mehdi Ali
* Seyithan Dag
* Mihail Luchian
* Muhammad Hassan Farooq
* Utkrist Adhikari
* Julian Tanke

### task 2.1 least squares regression for missing value prediction

Least squares polynomial fit with loss $E(w) = \big\vert\big\vert Xw -y \big\vert\big\vert^2$:

$$
w = \big( X^T X\big)^{-1} X^T y
$$

$$
X = \begin{bmatrix}
1 & x_1 & x_1^2 & \cdots  & x_1^d \\
 &  &  & \vdots  & \\
1 & x_n & x_n^2 & \cdots & x_n^d
\end{bmatrix}, \ \ \ \ y = \begin{bmatrix}
y_1 \\
\vdots \\
y_n\\
\end{bmatrix}
$$

For this task fit models for $d\in \{ 1, 5, 10\}$

```python
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
from numpy.linalg import inv, pinv

fig = plt.figure(figsize=(16,8))
ax = fig.add_subplot(111)

data = np.loadtxt('whData.dat',
                  dtype=np.object,
                  comments='#',
                  delimiter=None)

W = data[:,0].astype('float32')
H = data[:,1].astype('float32')

w_mask = ((W > 0) * 1).nonzero()
pred_mask = ((W < 0) * 1).nonzero()

H_to_predict = H[pred_mask]

W = W[w_mask]
H = H[w_mask]

# to make the code more numerically stable we down-scale it by some
# large number so that all values are between 0 and 1
scale = 200
W = W/scale
H = H/scale
H_to_predict = np.sort(H_to_predict/scale)

# plot the outliers
for pred in H_to_predict:
    ax.plot([pred, pred], [-10, 10], '--', color='black')

W = W.astype('float64')
H = H.astype('float64')

# solve the solution "zufuss"
# ------------ d = 1 ----------
X = np.vander(H, 1+1)  # (21,2)
y = np.array([W]).T    # (21,1)
w = inv(X.T @ X) @ (X.T @ y)
incl, intercpt = w
ax.plot([0, 200], [incl * 0 + intercpt, incl * 200 + intercpt],
         label='d = 1')

solve_d1 = lambda x: incl * x + intercpt

# ------------ d = 5 ----------
d = 5
X = np.vander(H, d+1)
y = np.array([W]).T        # (21,1)

# pinv is more numerically stable than the 'naive' implementation above
w = np.squeeze(pinv(X) @ y)  # pinv = (X^T X) X^T
_x_ = np.linspace( 0, 1, 10000)
p_d5 = np.poly1d(w)
ax.plot(_x_, p_d5(_x_), label="d = 5")

# ------------ d = 10 ----------
d = 10
X = np.vander(H, d+1)
y = np.array([W]).T        # (21,1)

# pinv is more numerically stable than the 'naive' implementation above
w = np.squeeze(pinv(X) @ y)  # pinv = (X^T X) X^T
_x_ = np.linspace( 0, 1, 10000)
p_d10 = np.poly1d(w)
ax.plot(_x_, p_d10(_x_), label="d = 10")

# ------------------------------

ax.scatter(H, W, color='black')
ax.set_xlabel('Height')
ax.set_ylabel('Weight')

ax.set_ylim([0/scale, 120/scale])
ax.set_xlim([155/scale, 190/scale])
ax.set_xticklabels([int(x * scale) for x in ax.get_xticks()])
ax.set_yticklabels([int(y * scale) for y in ax.get_yticks()])

print("d=5")
print("\t",H_to_predict*scale)
print("\t",p_d5(H_to_predict)*scale)

print("d=10")
print("\t",H_to_predict*scale)
print("\t",p_d10(H_to_predict)*scale)

ax.scatter(H_to_predict, solve_d1(H_to_predict), color='blue', s=80)
ax.scatter(H_to_predict, p_d5(H_to_predict), color='orange', s=80)
ax.scatter(H_to_predict, p_d10(H_to_predict), color='green', s=80)

plt.legend()
plt.show()
```

### task 2.2: conditional expectation for missing value prediction

Fit a bi-variate Gaussian to the height and weight data in $x$ and $y$ to model the joint density $p(x,y)$ of heights and weights.

Given your fitted bi-variate Gaussian, use the idea of conditional expectation to predict the weight values for the outliers. That is, let $x_0$ denote the available height data of an outlier and compute
$$
|\!E \big[ \ y \ \big| \ x_0 \ \big] =
\int y \ p(y | x_0 ) dy
$$

Do this either analytically as discussed in the lecture or numerically and report your results.

---

**Sample Covariance Matrix**: $Q=\frac{1}{n-1}\sum_{i=1}^n \big(
x_i - \overline{x}\big) (x_i - \overline{x})^T$

---

\begin{equation} \label{eq:gauss2d}
g(x,y) = A \ \mathrm{exp} \biggr[ - (a (x-x_0)^2+
2b(x-x_0)(y-y_0) + c(y-y_0)^2
\biggr]
\end{equation}

\begin{equation} \label{eq:gcovar2d}
\Sigma=\frac{1}{n-1}\sum_{i=1}^n \big(
x_i - \overline{x}\big) \big(x_i - \overline{x}\big)^T
= \begin{bmatrix}
a & b\\
b & c
\end{bmatrix}
\end{equation}

\begin{equation} \label{eq:gscale2d}
A = \frac{1}{(2\pi)^{\frac{D}{2}}\ \big\vert \ \Sigma \ \big\vert^{\frac{1}{2}}} \approx 0.0025 \ \ \ \ \mathrm{with} \ \ \ D=2
\end{equation}

```python
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
from numpy.linalg import inv, pinv
from scipy.stats import multivariate_normal

fig = plt.figure(figsize=(16,8))
ax = fig.add_subplot(111)

data = np.loadtxt('whData.dat',
                  dtype=np.object,
                  comments='#',
                  delimiter=None)

W = data[:,0].astype('float32')
H = data[:,1].astype('float32')
w_mask = ((W > 0) * 1).nonzero()
pred_mask = ((W < 0) * 1).nonzero()

H_to_predict = H[pred_mask]
W = W[w_mask]
H = H[w_mask]

ax.scatter(W, H, color='black')
ax.set_xlabel('Weight')
ax.set_ylabel('Height')
ax.set_ylim([120, 219])
ax.set_xlim([0, 119])

# --------------------------------
# mean
X = np.array([W, H])
mu = np.expand_dims(np.mean(X, 1), 1)  # expand so inner product works
ax.scatter(mu[0], mu[1], color='red')

# cov
_, n = X.shape
Q = (1/(n-1)) * ((X-mu) @ (X-mu).T)

F = multivariate_normal(np.squeeze(mu), Q)

# "integrate" numerically
steps = 10000
for h in H_to_predict:
    W_ = np.linspace(0, 220, steps)
    H_ = np.array([h] * steps)
    data = np.array([W_, H_]).T
    p = F.pdf(data)
    p = p/np.sum(p)
    result = np.sum(p * W_)
    ax.scatter(result, h, color='blue')
    print("Height " + str(h) + " yields " + str(result))

# plot
Im = np.zeros((220, 120))
for i,w in enumerate(range(0, 120)):  # welp, this sucks and could be
    for j,h in enumerate(range(0, 220)):  # better but who cares (I dont)
        Im[j,i] = F.pdf([w, h])

ax.matshow(Im)

# --------------------------------
plt.show()
```

### task 2.3: Baysian regression for missing value prediction
Use the method of Bayesian regression to fit a fifth degree polynomial $$y(x) = \sum_{j=0}^5w_jx^j$$
to he height and weight data in $x$ and $y$. Assume a Gaussian prior $$p(w) \sim \mathcal{N}\big( w \ \vert \ \mu_0, \sigma_0^2I \big)$$
for the parameter vector $w$ where $\mu_0=0$ and $\sigma_0^2=3$. Plot your resulting model and compare it to the corresponding model ($d=5$) from task 2.1

\begin{equation} \label{eq:map}
\theta_{\mathrm{MAP}} = \big(
X^T X + \frac{\sigma^2}{\sigma_0^2}I
\big)^{-1} \ X^T y
\end{equation}

```python
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib.pyplot as plt
from numpy.linalg import inv, pinv
from scipy.stats import multivariate_normal

# =============================
# loading data, ... etc ...
# =============================

fig = plt.figure(figsize=(16,8))
ax = fig.add_subplot(111)

data = np.loadtxt('whData.dat',
                  dtype=np.object,
                  comments='#',
                  delimiter=None)

W = data[:,0].astype('float32')
H = data[:,1].astype('float32')
w_mask = ((W > 0) * 1).nonzero()
pred_mask = ((W < 0) * 1).nonzero()

H_to_predict = H[pred_mask]
W = W[w_mask]
H = H[w_mask]

ax.scatter(H, W, color='black')
ax.set_ylabel('Weight')
ax.set_xlabel('Height')
ax.set_xlim([120, 219])
ax.set_ylim([0, 119])

# =============================
# starting
# =============================

s_sq = 1
s_sq_0 = 3
X = np.vander(H, 6, increasing=True)
y = W
n,k = X.shape
Reg = (s_sq/s_sq_0) * np.eye(k)

theta = inv(np.transpose(X) @ X + Reg) @ (np.transpose(X) @ y)

def func(h, theta):
    total = 0
    for p in range(6):
        total += h**p * theta[p]
    return total

H_plot = np.linspace(120, 220, 1000)
W_plot = [func(h, theta) for h in H_plot]

ax.plot(H_plot, W_plot, color='orange')

plt.show()
```


Series of N equals in coin tosses
======

For a fair coin, the question if it lands heads or tails is easily answered. Almost as simple is the question of what is the probability of having N heads (tails) in a row. However, it is not immediately obvious how we would answer a question like "How many tosses do we need to make before we will observe a series of N heads (tails)?". The answer is easily computed numerically and we will show an analytical solution in the end.

Numerical experiment
------

###Python imports

```python
import random as rd
import numpy as np
from numpy.random import choice
import matplotlib.pyplot as plt
import matplotlib

%matplotlib inline
matplotlib.style.use('ggplot')
matplotlib.rc_params_from_file("../styles/matplotlibrc" ).update()
```

### Constructing the experiment

It is obvious that to observe a series of N equals we need to have done N throws at least. Hence we initialize the experiment by tossing the coin N times. We define the categories as "H" and "T"

```python
list_of_categories = ["H", "T"]
```

```python
def initializeExperiment(N, prob = [0.5, 0.5]):
    tosses = []
    for idx in range(N):
        tosses.append(choice(list_of_categories, p = prob))
    return tosses
```

Next we need to check if the last N throws have been equal to the category we want to observe. To do this we construct a set of the last N tosses. If the size of the set is 1 and the category in the set is the one we are looking for we found a sequence of N equal tosses.

```python
def areLastNTossesEqualy(tosses, N, category):
    subset = set(tosses[-N:])
    if ((len(subset) == 1) and (category in subset)):
        return True
    else:
        return False
```

###Running the experiment

Since we have no prior knowledge of when the experiment will terminate we limit ourselves to a maximum number of tosses. We always check if the last N tosses have been heads (H). If yes, we terminate otherwise we continue with another toss.

```python
def runSingleExperiment(max_num_throws, number_of_equals, prob = [0.5,0.5]):
    tosses = initializeExperiment(number_of_equals, prob)
    throws = 0
    while throws < max_num_throws:
        if areLastNTossesEqualy(tosses, number_of_equals, "H"):
            return len(tosses)
        else:
            tosses.append(choice(list_of_categories))
        throws += 1
```

Finally we want to run M experiments and evaluate for the expected number of throws.

```python
def runKExperimentsAndEvaluate(m_experiments, number_of_equals, number_of_maximum_tosses=500, prob = [0.5,0.5]):
    number_of_tosses = []
    for idx in range(m_experiments):
        number_of_tosses.append(runSingleExperiment(number_of_maximum_tosses, number_of_equals, prob))

    return np.mean(number_of_tosses), np.std(number_of_tosses), number_of_tosses
```

So for 3 heads in a row, what's the expected number of tosses to observe this event"

```python
print "We expect to observe 3 heads after %3.2f tosses" % runKExperimentsAndEvaluate(5000, 3)[0]
```

    We expect to observe 3 heads after 13.97 tosses

As we will see later the non-integer nature of this expectation value is a residual of the numerical procedure we employed and it could easily be cast to an integer.
Before we get into the mathematical formulation of the problem, let's study the distribution a little bit more. A good way to gain some insight into the distribution is to look at the normalized histogram of the number of total throws until success. To this end we take a look at the raw tosses returned by `runKExperimentsAndEvaluate`

```python
tosses_three_equals = runKExperimentsAndEvaluate(25000, 3, number_of_maximum_tosses=1000)[2]
tosses_four_equals = runKExperimentsAndEvaluate(25000, 4, number_of_maximum_tosses=1000)[2]
tosses_five_equals = runKExperimentsAndEvaluate(25000, 5, number_of_maximum_tosses=1000)[2]
```

```python
bin_range = range(0,150, 2)
plt.hist(tosses_three_equals, bins=bin_range, normed=True)
plt.hist(tosses_four_equals, bins=bin_range, normed=True)
plt.hist(tosses_five_equals, bins=bin_range, normed=True)
plt.xlim([0,100])
```

Maybe surprisingly the distribution is not very well localized. In fact trying to fit it with an exponential function given the calculated mean fails. Increasing the number of required equals makes the curve flatter and more heavy tailed. Thus the variance itself is also large. In fact it is of the same order as the mean!

Mathematical calculation of the expectation value
------

Let us now turn towards the mathematical calculation of the expectation value. Note that we define success as the event $X$ where we have $n$ heads in a row. It is then easy to see that in whenever we observe tails we essentially waisted all tries before and need to start over again. However, we need to keep track of the wasted throws. If $p$ is the probability of having the coin land on head then the expectation value can be expressed simply by
\begin{align}
E(X) = np^n + (1-p) \left[E(X) +1\right] + p(1-p) \left[E(X) +2\right] +  p^2(1-p) \left[E(X) +3\right]...  = np^n + (1-p)\sum\limits_{i=0}^{n-1}p^i\left[E(X)+i+1\right)
\end{align}

Evaluating the sum we find the simple expression:
\begin{align}
E(X) = \frac{1-p^n}{p^n(1-p)}
\end{align}

To gain some understanding why above formula works let us look closer at the case for two eqaual tosses. In this case we have success if $X=HH$. To calculate the probability of this event happening we condition on the first toss, i.e. we calculate
\begin{align}
E(X) = E(X|H) p(H) + E(X|T) p(T)
\end{align}
where the condition is the result of the first toss. Expanding this some more we get
\begin{align}
E(X) = p^2 + (1-p) \left[E(X) + 1\right]
\end{align}
where we employed the same argument as above.

```python
def expectationValueForNumberOfTosses(p, number_of_equals):
    return int(float(1 - np.power(p, number_of_equals))/float(np.power(p, number_of_equals) * (1-p)))
```

```python
equals = np.linspace(1,20, 20)
y = []
for x in equals:
    y.append(expectationValueForNumberOfTosses(0.5, x))

plt.semilogy( equals, y, 'o')
```

From the plot aboe we see that the number of tosses until we have $N$ equals grows exponential! (Observe the logarithmic scale). For $N=20$ heads in a row we need on the order of 2 million successive throws for a fair coin. If we could manually throw a coin every second, it would take us about 23 days of uniterupted coin tossing until we could expect 20 heads in a row to appear... but remember: The distribution has a heavy tail and the variance is also on the order of 23 days!

Summary
------

We did some numerical experiments to investigate the question how many times we need to throw a coin before we can expect $N$ heads in a row. The distribution of this problem is surprisingly heavy tailed and becomes heavier the more heads are required. We saw that the variance is as large as the expectation value, which grows exponential with the number of required equals as we proved mathematically.

Enjoy this notebook! Contributions and/or feedback are welcome as always!

###Stylesheet

```python
from IPython.core.display import HTML

def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```python

```

Numerical-Experimentation/Series of N equals in coin tosses.ipynb


##Master Theorem

$$ T(n) = aT(\frac{n}{b}) + O(n^k) $$

So...

$$
T(n) = \begin{equation}
\left\{
  \begin{aligned}
    O(n^k)           && a  b^k
  \end{aligned}
  \right.
\end{equation}
$$

```python
import random

lst = list(range(10))
random.shuffle(lst)
print(lst)
```

    [9, 3, 8, 1, 4, 7, 2, 5, 0, 6]

##Multiplicacion de dos numeros

```python
def mult(a, b, n):
  if n == 1:
    return a * b

  ai = a // 10**(n//2)
  ad = a % 10**(n//2)
  bi = b // 10**(n//2)
  bd = b % 10**(n//2)
  z1 = mult(ai, bi, n//2) * 10**n
  z2 = (mult(ai, bd, n//2) + mult(ad, bi, n//2)) * 10**(n//2)
  z3 = mult(ad, bd, n//2)

  return z1 + z2 + z3

a = 548929495
b = 102043934
n = 16
res = mult(a, b, n)
print(res)
```

    56014925158433330

##El mayor de un arreglo

###Fuerza Bruta

```python
def bfmax(a):
  m = a[0]
  for i in range(1, len(a)):
    if a[i] > m:
      m = a[i]
  return m
```

```python
bfmax(lst)
```

    9

### Divide y venceras

```python
def dcmax(a, i, f):
  if i == f:
    return a[i]

  mid = (i + f) // 2
  maxleft = dcmax(a, i, mid)
  maxright = dcmax(a, mid + 1, f)

  return maxleft if maxleft > maxright else maxright

dcmax(lst, 0, len(lst)-1)
```

    9

$$ T(n) = 2T(\frac{n}{2}) + O(1) $$

$$
a = 2\\
b = 2\\
k = 0
$$

$$
T(n) = n^{\log_2{2}}\\
T(n) = n\\
T(n) \implies O(n)
$$

##Sumatoria de un arreglo

```python
def suma(ini, fin, arr):
  if (ini == fin):
    return arr[ini]
  else:
    mitad = (ini+fin)//2
    x = suma(ini, mitad, arr)
    y = suma(mitad+1, fin,arr)
    return x+y

suma(0,len(lst)-1, lst)
```

    45

##Multiplicacion de matrices

para matrices donde $n = 2^k \,,\, k \in N$

### Classic way

```python
def matmul(a, b):
  n = len(a)
  c = [[0]*n for _ in range(n)]
  for i in range(n):
    for j in range(n):
      accum = 0
      for k in range(n):
        accum += a[i][k] * b[k][j]
      c[i][j] = accum
  return c
```

```python
import numpy as np
```

```python
n = 8
a = np.array(list(range(n*n))).reshape((n, n))
b = np.array(list(range(n*n))).reshape((n, n))
print(a)
print(b)
```

    [[ 0  1  2  3  4  5  6  7]
     [ 8  9 10 11 12 13 14 15]
     [16 17 18 19 20 21 22 23]
     [24 25 26 27 28 29 30 31]
     [32 33 34 35 36 37 38 39]
     [40 41 42 43 44 45 46 47]
     [48 49 50 51 52 53 54 55]
     [56 57 58 59 60 61 62 63]]
    [[ 0  1  2  3  4  5  6  7]
     [ 8  9 10 11 12 13 14 15]
     [16 17 18 19 20 21 22 23]
     [24 25 26 27 28 29 30 31]
     [32 33 34 35 36 37 38 39]
     [40 41 42 43 44 45 46 47]
     [48 49 50 51 52 53 54 55]
     [56 57 58 59 60 61 62 63]]

```python
c = matmul(a, b)
np.array(c)
```

    array([[ 1120,  1148,  1176,  1204,  1232,  1260,  1288,  1316],
           [ 2912,  3004,  3096,  3188,  3280,  3372,  3464,  3556],
           [ 4704,  4860,  5016,  5172,  5328,  5484,  5640,  5796],
           [ 6496,  6716,  6936,  7156,  7376,  7596,  7816,  8036],
           [ 8288,  8572,  8856,  9140,  9424,  9708,  9992, 10276],
           [10080, 10428, 10776, 11124, 11472, 11820, 12168, 12516],
           [11872, 12284, 12696, 13108, 13520, 13932, 14344, 14756],
           [13664, 14140, 14616, 15092, 15568, 16044, 16520, 16996]])

### Divide and conquer way

```python
def dcmatmul(a, b, c, ri, rf, ci, cf):
  n = len(a)
  if ri == rf:
    accum = 0
    for k in range(n):
      accum += a[ri][k] * b[k][cf]

    c[ri][cf] = accum
  else:
    rmid = (ri + rf) // 2
    cmid = (ci + cf) // 2
    dcmatmul(a, b, c, ri, rmid, ci, cmid)
    dcmatmul(a, b, c, rmid+1, rf, ci, cmid)
    dcmatmul(a, b, c, ri, rmid, cmid +1, cf)
    dcmatmul(a, b, c, rmid+1, rf, cmid + 1, cf)
```

```python
n = 8
a = np.array(list(range(n*n))).reshape((n, n))
b = np.array(list(range(n*n))).reshape((n, n))
c = np.zeros((n, n))
dcmatmul(a, b, c, 0, n-1, 0, n-1)
print(c)
```

    [[ 1120.  1148.  1176.  1204.  1232.  1260.  1288.  1316.]
     [ 2912.  3004.  3096.  3188.  3280.  3372.  3464.  3556.]
     [ 4704.  4860.  5016.  5172.  5328.  5484.  5640.  5796.]
     [ 6496.  6716.  6936.  7156.  7376.  7596.  7816.  8036.]
     [ 8288.  8572.  8856.  9140.  9424.  9708.  9992. 10276.]
     [10080. 10428. 10776. 11124. 11472. 11820. 12168. 12516.]
     [11872. 12284. 12696. 13108. 13520. 13932. 14344. 14756.]
     [13664. 14140. 14616. 15092. 15568. 16044. 16520. 16996.]]


# Computing derivatives with Fourier transforms

```python
import matplotlib.pyplot as plt
import numpy as np
import os
import sys
sys.path.append(os.path.join(os.getcwd(), '../build/'))
import pydeft as deft

plt.rc('font', family='serif')
plt.rc('xtick', labelsize='x-small')
plt.rc('ytick', labelsize='x-small')
```

### Computing the gradient

The gradient of a function and its Fourier transform are related by

\begin{equation}
\nabla w(\mathbf{r}) \xrightarrow[\text{Fourier transform}]{} i \mathbf{k} \tilde{w}(\mathbf{k}) .
\end{equation}

The following code uses DEFT to compute the gradient of $w(\mathbf{r}) = \cos(4\pi x)$, defined on a simple cubic lattice with unit spacing. The result should be $-4\pi\sin(4\pi x)$ along the $x$-direction and zero along the $y$- and $z$- directions.

```python
# create box and data array
box_vectors = np.eye(3)
grd_pts = 25
box = deft.Box(box_vectors)
data = deft.Double3D([grd_pts, grd_pts, grd_pts])

# get f(r) = cos(4*pi*x) on grid
for i in range(grd_pts):
    data[i,:,:] = np.cos(4.0*np.pi*float(i)/grd_pts)

# compute gradients
grad_x = deft.gradient_x(data, box)
grad_y = deft.gradient_y(data, box)
grad_z = deft.gradient_z(data, box)

# plot
x = np.linspace(0,1,500)
fig = plt.figure(figsize=[9,3.5])
plt.plot(x, -4.0*np.pi*np.sin(4.0*np.pi*x), color='0.5', ls = 'solid', linewidth = 2)
plt.plot(np.arange(grd_pts)/grd_pts, grad_x[:,0,0], '.k', markersize = 8)
plt.plot(np.arange(grd_pts)/grd_pts, grad_y[:,0,0], '+k', markersize = 8)
plt.plot(np.arange(grd_pts)/grd_pts, grad_z[:,0,0], 'xk', markersize = 8)
plt.xlim([-0.01, 1.01])
plt.xlabel('$x$-axis [one lattice cell]')
plt.legend(['$-4\pi\sin(4\pi x)$', '$\\nabla_x w(\mathbf{r})$', '$\\nabla_y w(\mathbf{r})$', '$\\nabla_z w(\mathbf{r})$'], loc=(1.05,0.25))
plt.tight_layout()
plt.show()
```

### Computing the Laplacian

The Laplacian of a function and its Fourier transform are related by

\begin{equation}
\nabla^2 w(\mathbf{r})
\xrightarrow[\text{Fourier transform}]{}
- k^2 \tilde{w}(\mathbf{k}) .
\end{equation}

The following code uses DEFT to compute the Laplacian of $w(\mathbf{r}) = \cos(4\pi x)$, defined on a simple cubic lattice with unit spacing. The result should be $-16\pi^2\cos(4\pi x)$.

```python
# create box and data array
box_vectors = np.eye(3)
grd_pts = 25
box = deft.Box(box_vectors)
data = deft.Double3D([grd_pts, grd_pts, grd_pts])

# get f(r) = cos(4*pi*x) on grid
for i in range(grd_pts):
    data[i,:,:] = np.cos(4.0*np.pi*float(i)/grd_pts)

# compute laplacian
lapl = deft.laplacian(data, box)

# plot
x = np.linspace(0,1,500)
plt.rc('font', size=14)
fig = plt.figure(figsize=[9,3.5])
plt.plot(x, -16.0*np.pi**2*np.cos(4.0*np.pi*x), color='0.5', ls = 'solid', linewidth = 2)
plt.plot(np.arange(grd_pts)/grd_pts, lapl[:,0,0], '.k', markersize = 8)
plt.xlim([-0.01, 1.01])
plt.xlabel('$x$-axis [one lattice cell]')
plt.legend(['$-16\pi^2\cos(4\pi x)$', '$\\nabla^2 w(\mathbf{r})$'], loc=(1.03,0.4))
plt.tight_layout()
plt.show()
```