# Introduction

In this note we

* Demonstrate the basic steps needed to set up the Poisson problem
* Solve the problem
* Demonstrate how these steps can be put inside the `tf.keras` functional API

```python
import numpy as np
import tensorflow as tf
tf.keras.backend.set_floatx('float64')
import tenfem
import matplotlib.pyplot as plt
from matplotlib.collections import LineCollection
from matplotlib.tri import Triangulation
from tenfem.layers import (AssembleStiffnessMatrix, AssembleLoadVector, SolveDirichletProblem,
                           TriangleMeshInterpolator)

tf_dtype = tf.float64
tfk = tf.keras
tfkl = tf.keras.layers
```

# The Poisson Problem

The Poisson problem is to PDE problems, and the FEM, what `Hello, world` is to coding exercises. Mathematically
we are interested in the following *boundary value problem*

$$
\begin{align}
    -\Delta u(x) &= f(x), \qquad x \in \Omega \tag{1a} \\
    u(x) &= u_{D}(x) \qquad x \mbox{ on } \partial\Omega. \tag{1b}
\end{align}
$$

The goal is to find the unknown function $u(x)$ which solves this problem, that is for any $x$ inside
the domain $\Omega$ we satisfy (1a), and for any $x$ on the boundary $\partial \Omega$ we satisfy
(1b). Here the boundary condition specified by the function $u_{D}$ is referred to as a Dirichlet
boundary condition, and the function $f(x)$ is typically referred to as the source or sink function.

If we want to implement such a problem then we need to specify our domain $\Omega$, and so this is where
we go next.

## A `TriangleMesh` of $\Omega$

All of the domains will be specifed by decomising a subset $\Omega \subset \mathbb{R}^D$ into a collection
of disjoint polygonal regions sharing edges -- the eponymous *elements* of the finite element method. Below
we show an example which contains a regular tesselation of the unit square $\Omega = [0, 1] \times [0, 1]$
by triangle elements, this is an instance of a `tenfem.mesh.TriangleMesh` object.

```python
mesh = tenfem.mesh.examples.square(8, 8, dtype=tf_dtype)

fig, ax = plt.subplots()
ax.triplot(mesh._triang)

ax.add_collection(
    LineCollection(tf.gather(mesh.nodes, mesh.boundary_elements), colors='k', label='$\partial \Omega$'))
ax.legend(fontsize=14)
ax.set_aspect('equal'); ax.axis('off')
for method in [ax.set_xlim, ax.set_ylim]:
    method((-0.1, 1.1))

plt.show()
```

One of the beautiful aspects of the Finite Element Method is that while the original problem
over the domain $\Omega$ is specified in terms of the "physical elements" -- our triangles
in the mesh above. We solve the equation by considering each of these physical elements as
a transformation of a single canonical *reference element*, with a bijection between the
physical elements and the canonical element, this abstraction is encapsulated by the
`tenfem.reference_element.BaseReferenceElement` class.

## `AssembleStiffnessMatrix`

```python
lin_tri_element = tenfem.reference_elements.TriangleElement(degree=1, dtype=tf_dtype)
mesh = tenfem.mesh.examples.square(16, 16, dtype=tf_dtype)  # lets make a higher res mesh to see the solution

diffusion_coefficient = lambda x: tf.ones_like(x)[..., 0]
source = lambda x: -6. * tf.ones_like(x)[..., 0]
ud = lambda x: (1 + x[...,0]**2 + 2. * x[..., 1]**2)[..., tf.newaxis]

def solve(mesh, element):
    mesh_tensor_repr = mesh.get_tensor_repr()

    stiffness_matrix = AssembleStiffnessMatrix(diffusion_coefficient,
                                               reference_element=element)(mesh_tensor_repr)
    load_vector = AssembleLoadVector(source,
                                     reference_element=element)(mesh_tensor_repr)
    node_types = mesh_tensor_repr[-1]
    boundary_values = ud(tf.gather_nd(mesh.nodes, tf.where(node_types == 1)))

    return tenfem.fem.solve_dirichlet_form_linear_system(
        stiffness_matrix[0], load_vector[0], node_types, boundary_values)
```

```python
solution = solve(mesh, lin_tri_element)
solution_exact = ud(mesh.nodes)

err = np.max(np.abs(solution - solution_exact))
```

```python
fig, axes = plt.subplots(ncols=3, figsize=(10, 6))

axes[0].tricontourf(mesh._triang, solution[:, 0], levels=20)
axes[1].tricontourf(mesh._triang, ud(mesh.nodes)[:, 0], levels=20)
axes[2].tricontourf(mesh._triang, np.abs(solution[:, 0] - solution_exact[:, 0]))

titles = ['Numerical', 'Exact', 'Abs. Error (max = {:.2e})'.format(err)]

for i, ax in enumerate(axes):
    ax.add_collection(
        LineCollection(tf.gather(mesh.nodes, mesh.boundary_elements), colors='k', label='$\partial \Omega$'))
    ax.set_aspect('equal'); ax.axis('off')
    for method in [ax.set_xlim, ax.set_ylim]:
        method((-0.1, 1.1))
    ax.set_title(titles[i])

plt.show()
```

For statistical applications and error of the magnitude above might be acceptable,
however this is less likely to be the case in engineering applications! How can
we decrease this error? One option is to specify a mesh with a finder resolution,
alternatively we can try and perform the numerical quadratures more accurately.

One of the most important ways this manifests is that whenever we are doing integration
over a physical element, we parameterise in terms of coordinates of the reference element,
and in practice specify integrations by quadratures over the reference element. Once
we have specified a quadrature order, all of the callable components of our model will be
queried for values at these quadrature nodes, so lets inspect them.

We provide a utility function to convert a standard triangular mesh to one suitable
for degree 2 basis functions in the `tenfem.mesh.triangle.convert_linear_to_quadratic`
function, and we use this to increase the degree of our approximations on the mesh, and
so we will also have to change the degree of the `TriangleElement` reference element
we are using.

```python
quad_mesh = tenfem.mesh.triangle.convert_linear_to_quadratic(mesh)
quad_tri_element = tenfem.reference_elements.TriangleElement(degree=2, dtype=tf_dtype)

elements = [lin_tri_element, quad_tri_element]
meshes = [mesh, quad_mesh]

fig, axes = plt.subplots(ncols=2, figsize=(8, 6))
for i, ax in enumerate(axes):

    quadrature_nodes = elements[i].get_quadrature_nodes(meshes[i])

    ax.triplot(mesh._triang)

    ax.plot(*tf.reshape(quadrature_nodes, [-1, 2]).numpy().T,
            'o', markeredgecolor='k', label='quadrature nodes' if i == 0 else None)
    ax.add_collection(
        LineCollection(tf.gather(mesh.nodes, mesh.boundary_elements), colors='k'))

    if i == 0:
        ax.legend(fontsize=14, loc='lower left')

    ax.set_aspect('equal'); ax.axis('off')
    for method in [ax.set_xlim, ax.set_ylim]:
        method((0.83, 1.01))

    ax.set_title('Degree = {}'.format(i+1))
plt.show()
```

We can see that the actualy number of elements hasn't changed, but
that while before we approximated each one with 3 quadrature nodes
we now use 6. As such the total number of nodes over which we need
to solve the FEM problem has increased.

Now that we have increase the degree of our basis functions on the
mesh elements lets solve the problem, and then plot the results

```python
sol_deg2 = solve(quad_mesh, quad_tri_element)
solution_exact = ud(quad_mesh.nodes)
err = np.max(np.abs(sol_deg2 - solution_exact))
```

```python
mtri = Triangulation(*quad_mesh.nodes.numpy().T)

fig, axes = plt.subplots(ncols=3, figsize=(10, 6))

axes[0].tricontourf(mtri, sol_deg2[:, 0], levels=20)
axes[1].tricontourf(mtri, ud(quad_mesh.nodes)[:, 0], levels=20)
axes[2].tricontourf(mtri, np.abs(sol_deg2[:, 0] - solution_exact[:, 0]))

titles = ['Numerical' % err, 'Exact', 'Abs. Error, (max = {:.2e})'.format(err)]

for i, ax in enumerate(axes):
    ax.add_collection(
        LineCollection(tf.gather(mesh.nodes, mesh.boundary_elements), colors='k', label='$\partial \Omega$'))
    ax.set_aspect('equal'); ax.axis('off')
    for method in [ax.set_xlim, ax.set_ylim]:
        method((-0.1, 1.1))
    ax.set_title(titles[i])
```

# PDE inside of a `tf.keras.Model`

```python
mesh_provider = tenfem.layers.MeshProvider(quad_mesh,
                                           quad_tri_element)
source = tfkl.Dense(1)
#source = lambda x: -6. * tf.ones_like(x)[..., 0]

inputs = tfkl.Input(2, dtype=tf_dtype)
mesh_tensor_repr = mesh_provider(inputs)

stiffness_matrix = AssembleStiffnessMatrix(
    diffusion_coefficient,
    reference_element=mesh_provider.reference_element)(mesh_tensor_repr)

load_vector = AssembleLoadVector(source,
                                 reference_element=mesh_provider.reference_element,
                                 name='assemble_load_vector')(mesh_tensor_repr)

u = SolveDirichletProblem(ud,
                          reference_element=mesh_provider.reference_element)(
    [stiffness_matrix,
     load_vector,
     mesh_tensor_repr])

ux = TriangleMeshInterpolator(tri_finder=mesh._triang.get_trifinder(),
                              reference_element=mesh_provider.reference_element)(
    [inputs, u, mesh_tensor_repr])

model = tfk.Model(inputs=inputs, outputs=ux)
model.summary()

```

    Model: "model"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_1 (InputLayer)            [(None, 2)]          0
    __________________________________________________________________________________________________
    mesh_provider (MeshProvider)    ((961, 2), (450, 6), 0           input_1[0][0]
    __________________________________________________________________________________________________
    assemble_stiffness_matrix (Asse (1, 961, 961)        0           mesh_provider[0][0]
                                                                     mesh_provider[0][1]
                                                                     mesh_provider[0][2]
                                                                     mesh_provider[0][3]
    __________________________________________________________________________________________________
    assemble_load_vector (AssembleL (1, 961, 1)          3           mesh_provider[0][0]
                                                                     mesh_provider[0][1]
                                                                     mesh_provider[0][2]
                                                                     mesh_provider[0][3]
    __________________________________________________________________________________________________
    solve_dirichlet_problem (SolveD (961, 1)             0           assemble_stiffness_matrix[0][0]
                                                                     assemble_load_vector[0][0]
                                                                     mesh_provider[0][0]
                                                                     mesh_provider[0][1]
                                                                     mesh_provider[0][2]
                                                                     mesh_provider[0][3]
    __________________________________________________________________________________________________
    trimesh_interpolate (TriangleMe None                 0           input_1[0][0]
                                                                     solve_dirichlet_problem[0][0]
                                                                     mesh_provider[0][0]
                                                                     mesh_provider[0][1]
                                                                     mesh_provider[0][2]
                                                                     mesh_provider[0][3]
    ==================================================================================================
    Total params: 3
    Trainable params: 3
    Non-trainable params: 0
    __________________________________________________________________________________________________

```python
inputs = np.random.uniform(size=100*2).reshape(100, 2)
targets = ud(inputs)

model.compile(optimizer='rmsprop', loss='mean_squared_error')
history = model.fit(inputs, targets, epochs=100)
```

    Epoch 1/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0737
    Epoch 2/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0734
    Epoch 3/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0732
    Epoch 4/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0730
    Epoch 5/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0728
    Epoch 6/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0726
    Epoch 7/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0724
    Epoch 8/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0722
    Epoch 9/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0721
    Epoch 10/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0719
    Epoch 11/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0717
    Epoch 12/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0715
    Epoch 13/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0713
    Epoch 14/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0712
    Epoch 15/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0710
    Epoch 16/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0708
    Epoch 17/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0707
    Epoch 18/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0705
    Epoch 19/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0704
    Epoch 20/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0702
    Epoch 21/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0700
    Epoch 22/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0698
    Epoch 23/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0697
    Epoch 24/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0695
    Epoch 25/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0694
    Epoch 26/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0692
    Epoch 27/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0690
    Epoch 28/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0688
    Epoch 29/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0687
    Epoch 30/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0685
    Epoch 31/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0683
    Epoch 32/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0681
    Epoch 33/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0680
    Epoch 34/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0678
    Epoch 35/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0677
    Epoch 36/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0675
    Epoch 37/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0673
    Epoch 38/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0672
    Epoch 39/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0670
    Epoch 40/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0669
    Epoch 41/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0667
    Epoch 42/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0665
    Epoch 43/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0664
    Epoch 44/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0662
    Epoch 45/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0660
    Epoch 46/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0658
    Epoch 47/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0657
    Epoch 48/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0655
    Epoch 49/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0653
    Epoch 50/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0651
    Epoch 51/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0650
    Epoch 52/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0649
    Epoch 53/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0647
    Epoch 54/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0646
    Epoch 55/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0644
    Epoch 56/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0642
    Epoch 57/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0640
    Epoch 58/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0639
    Epoch 59/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0637
    Epoch 60/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0636
    Epoch 61/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0634
    Epoch 62/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0632
    Epoch 63/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0631
    Epoch 64/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0629
    Epoch 65/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0628
    Epoch 66/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0626
    Epoch 67/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0625
    Epoch 68/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0623
    Epoch 69/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0622
    Epoch 70/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0620
    Epoch 71/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0618
    Epoch 72/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0617
    Epoch 73/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0615
    Epoch 74/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0614
    Epoch 75/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0612
    Epoch 76/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0610
    Epoch 77/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0609
    Epoch 78/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0607
    Epoch 79/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0606
    Epoch 80/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0604
    Epoch 81/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0602
    Epoch 82/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0601
    Epoch 83/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0599
    Epoch 84/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0598
    Epoch 85/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0597
    Epoch 86/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0595
    Epoch 87/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0593
    Epoch 88/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0592
    Epoch 89/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0590
    Epoch 90/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0589
    Epoch 91/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0587
    Epoch 92/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0585
    Epoch 93/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0584
    Epoch 94/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0583
    Epoch 95/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0581
    Epoch 96/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0580
    Epoch 97/100
    4/4 [==============================] - 0s 30ms/step - loss: 0.0578
    Epoch 98/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0576
    Epoch 99/100
    4/4 [==============================] - 0s 29ms/step - loss: 0.0575
    Epoch 100/100
    4/4 [==============================] - 0s 28ms/step - loss: 0.0573

```python

```


```python
import cmath, random, numpy
import functools
import  matplotlib.pyplot as plt
import sys
import os
import math
from qutip import*

from sympy import*
#from sympsi import*
from scipy import optimize

from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
import time
import math
from qutip import *
from qutip.ipynbtools import plot_animation
import numpy as np
import matplotlib.pyplot as plt
import qutip
%matplotlib inline
import matplotlib.pylab as plt
import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from IPython.display import display, Math, Latex
import cmath
from mpl_toolkits.axes_grid1 import AxesGrid
from scipy.special import factorial
```

```python
"""Define the operators for MZI - we will deal in Heisenberg """
T = Symbol('T')
xvec = np.arange(-40.,40.)*5./40
yvec = np.arange(-50.,50)*5/40
X,Y = np.meshgrid(xvec, xvec)  ##Some plotting params
X1,Y1 = np.meshgrid(yvec,yvec)
N_dim = 35##Dimenstion of the Hilbert spac

a1 = destroy(N_dim) ##This is for single-photon field
a2 = destroy(N_dim) ##for coherent field
a3 = destroy(N_dim) ##for vacuum field
def n_choose_k(n,k):
    return factorial(n)/(factorial(n-k)*factorial(k))
def D(state,alpha):
    Rho_new=displace(N_dim,alpha)*state*displace(N_dim,alpha).dag()
    return Rho_new
'''Define a rotation in phase space, or phase shifter operation'''
def Phase(theta):
    b=-1j*theta*a1.dag()*a1;
    return b.expm()
'''Squeezing operation, inputs a density matrix and outputs the squeezed density matrix for squeezing parameter r'''
def Sq(state,r):
    Rho_new=squeeze(N_dim,r)*state*squeeze(N_dim,r).dag();
    return Rho_new
'''The function below creates a beamsplitter operation that acts on two modes.
The value for k determines what number Fock state could be filtered out of the first state
based on a single photon input for the second BS port, followed by single photon detection.'''
def BS_operator_filtering(a1, a2, k):
    theta_k = np.arctan(1/np.sqrt(k))
    T = np.sin(theta_k)*np.sin(theta_k)
    R = np.cos(theta_k)*np.cos(theta_k)
    print('I am filtering', k, 'and:', theta_k*180/math.pi)
    print('BS T is : ', T, 'and : ', R)
    b = theta_k*(tensor(a1,a2.dag()) - tensor(a1.dag(),a2))
    return b.expm()
def SSV_plus(r,alpha):
    state = ket2dm((displace(N_dim,alpha)+displace(N_dim,-alpha))*squeeze(N_dim,r)*fock(N_dim,0))
    norm_state = state/state.tr()
    return norm_state
def SSV_minus(r,alpha):
    state = ket2dm((displace(N_dim,alpha)-displace(N_dim,-alpha))*squeeze(N_dim,r)*fock(N_dim,0))
    norm_state = state/state.tr()
    return norm_state
def cat_plus(alpha):
    cat = (1/(np.sqrt(2)*np.sqrt(1+np.e**(-alpha*alpha.conj()))))*(coherent(N_dim,-alpha)+(coherent(N_dim,alpha)))
    return cat
def cat_minus(alpha):
    cat = (1/(np.sqrt(2)*np.sqrt(1-np.e**(-alpha*alpha.conj()))))*(-coherent(N_dim,-alpha)+(coherent(N_dim,alpha)))
    return cat
def pnr_resolution_detector(eta, click, n_truc):
    pi_n = 0;
    l = np.arange(click,n_truc)
    for i in l:
        pi_n +=  n_choose_k(i,click)*math.pow((1-eta),(i-click))*math.pow(eta,click)*fock(N_dim,i)*fock(N_dim,i).dag()
        #print("The final Povm element is: ", pi_0)
    return Qobj(pi_n)
def Fock_Filter_povm(in_state,in_fock,refl,num_det,eta,n_truc):
    Projector = tensor(pnr_resolution_detector(eta, num_det, n_truc),qeye(N_dim));
    Initial_state=tensor(in_state,ket2dm(fock(N_dim,in_fock)));
    theta_k=np.arccos(np.sqrt(refl));

    BS1= ((theta_k)*(tensor(a1,a2.dag()) - tensor(a1.dag(),a2))).expm()

    Rho=BS1*Initial_state*BS1.dag();

    Rho_filtered = ((Rho*Projector).ptrace(1))/((Rho*Projector).tr())
    '''The operation .ptrace(m) takes the partial trace over every mode EXCEPT m, where the numbering
    startes at 0.  So .ptrace(1) means you keep mode 1, which is actually the 2nd mode'''
    print('BS has reflectivity',refl,' and I am detecting the |',num,'> state, where my detector has efficiency', eta)
    return Rho_filtered
def Fock_Filter_prob(in_state,in_fock,refl,num_det,eta,n_truc):
    Projector = tensor(pnr_resolution_detector(eta, num_det, n_truc),qeye(N_dim));
    Initial_state=tensor(in_state,ket2dm(fock(N_dim,in_fock)));
    theta_k=np.arccos(np.sqrt(refl));

    BS1= ((theta_k)*(tensor(a1,a2.dag()) - tensor(a1.dag(),a2))).expm()

    Rho=BS1*Initial_state*BS1.dag();
    P=(Rho*Projector).tr()
    print('The probability of a sucessful detection is:',P)
    Rho_filtered = ((Rho*Projector).ptrace(1))/((Rho*Projector).tr())
    #Rho_filtered=Rho*Projector
    '''The operation .ptrace(m) takes the partial trace over every mode EXCEPT m, where the numbering
    startes at 0.  So .ptrace(1) means you keep mode 1, which is actually the 2nd mode'''
    print('BS has reflectivity',refl,' and I am detecting the |',num,'> state, where my detector has efficiency', eta)
    return Rho_filtered

def fid(state1,state2):
    F=np.absolute((state1.sqrtm()*state2*state1.sqrtm()).sqrtm().tr())
    return F

```

```python
#Variable definitions:
#delta = initial coherent state amplitude
# refl(1-4)= beamsplitter r^2 values at each step
# n(1-4) = number of photons detected at each detector
# beta = amplitude of final dispalcement to displace the state back
#alpha = amplitude of SSV state to be compared with.  Note, here
#this can be related to cat state amplitude by amp_cat=alpha/(Cosh[sq]-Sinh[sq])
# sq = the 'r' value, or squeezing parameter. This is the negative of the mathematica results.

'''Four-step check: r[4.] = 0.42134; r[3.] = 0.69684; r[2.] = 0.55398;
r[1.] = .576813; \[Delta] = 4.6621868; n[4.] = 1; n[3.] = 2;
n[2.] = 4; n[1.] = 6;
sq= -0.51, alpha= 1.59, beta= 2.08625'''

refl1= .576813**2; refl2=0.55398**2; refl3=0.69684**2;refl4=0.42134**2;
delta=4.6621868; beta=2.08625; sq=0.476595; alpha=1.59; n1=6;n2=4;n3=2;n4=1;
```

```python
ssv=SSV_plus(sq,alpha)

W_ssv=wigner(ssv,xvec,xvec);
eta=1;

first=Fock_Filter_prob(ket2dm(coherent(N_dim,delta)),1,refl1,n1,eta,N_dim)
W1=wigner(first,yvec,yvec);

fig = plt.figure(figsize=(16,7))
# `ax` is a 3D-aware axis instance, because of the projection='3d' keyword argument to add_subplot
ax = fig.add_subplot(2, 2, 2)
p = ax.contourf(X, Y, W_ssv, 20, cmap=cm.RdBu)
cb = fig.colorbar(p, shrink = 0.7)
# surface_plot with color grading and color bar
ax = fig.add_subplot(2, 2, 1, projection='3d')
p = ax.plot_surface(X, Y, W_ssv, rstride=1, cstride=1, cmap=cm.RdBu, linewidth=0.5)
cb = fig.colorbar(p,shrink = .7)
plt.title('Wigner function of ssv state')

fig = plt.figure(figsize=(16,7))
# `ax` is a 3D-aware axis instance, because of the projection='3d' keyword argument to add_subplot
ax = fig.add_subplot(2, 2, 2)
p = ax.contourf(X1, Y1, W1, 20, cmap=cm.RdBu)
cb = fig.colorbar(p, shrink = 0.7)
# surface_plot with color grading and color bar
ax = fig.add_subplot(2, 2, 1, projection='3d')
p = ax.plot_surface(X1, Y1, W1, rstride=1, cstride=1, cmap=cm.RdBu, linewidth=0.5)
cb = fig.colorbar(p,shrink = .7)
plt.title('Wigner function of filtered state')
fid = fidelity(ssv,first)
print('fidelity with ssv:',fid*fid)

fig, axes = plt.subplots(1, 3, figsize=(12,4))
bar0 = axes[0].bar(range(N_dim), first.diag())

lbl0 = axes[0].set_title("Filtered State")

lim0 = axes[0].set_xlim([-.5, 20])
bar1 = axes[1].bar(range(N_dim), ssv.diag())

lbl1 = axes[1].set_title("SSV")

lim1 = axes[1].set_xlim([-.5, 20])

bar2 = axes[2].bar(range(N_dim), ket2dm(coherent(N_dim,delta)).diag())

lbl2 = axes[2].set_title("Initial Coherent State")

lim2 = axes[2].set_xlim([-.5, 20])

plt.show()
```

```python
second=Fock_Filter_prob(first,1,refl2,n2,eta,N_dim);
W2=wigner(second,yvec,yvec);

fig = plt.figure(figsize=(16,7))
# `ax` is a 3D-aware axis instance, because of the projection='3d' keyword argument to add_subplot
ax = fig.add_subplot(2, 2, 2)
p = ax.contourf(X1, Y1, W2, 20, cmap=cm.RdBu)
cb = fig.colorbar(p, shrink = 0.7)
# surface_plot with color grading and color bar
ax = fig.add_subplot(2, 2, 1, projection='3d')
p = ax.plot_surface(X1, Y1, W2, rstride=1, cstride=1, cmap=cm.RdBu, linewidth=0.5)
cb = fig.colorbar(p,shrink = .7)
plt.title('Wigner function of filtered state')
fid = fidelity(ssv,second)
print('fidelity with cat:',fid*fid)

fig, axes = plt.subplots(1, 3, figsize=(12,4))
bar0 = axes[0].bar(range(N_dim), second.diag())

lbl0 = axes[0].set_title("Twice Filtered State")

lim0 = axes[0].set_xlim([-.5, 20])
bar1 = axes[1].bar(range(N_dim), first.diag())

lbl1 = axes[1].set_title("First Filtered State")

lim1 = axes[1].set_xlim([-.5, 20])

bar2 = axes[2].bar(range(N_dim), ket2dm(coherent(N_dim,delta)).diag())

lbl2 = axes[2].set_title("Initial Coherent State")

lim2 = axes[2].set_xlim([-.5, 20])
plt.show()
```

```python
third=Fock_Filter_prob(second,1,refl3,n3,eta,N_dim);
W3=wigner(third,yvec,yvec);

fig = plt.figure(figsize=(16,7))
# `ax` is a 3D-aware axis instance, because of the projection='3d' keyword argument to add_subplot
ax = fig.add_subplot(2, 2, 2)
p = ax.contourf(X1, Y1, W3, 20, cmap=cm.RdBu)
cb = fig.colorbar(p, shrink = 0.7)
# surface_plot with color grading and color bar
ax = fig.add_subplot(2, 2, 1, projection='3d')
p = ax.plot_surface(X1, Y1, W3, rstride=1, cstride=1, cmap=cm.RdBu, linewidth=0.5)
cb = fig.colorbar(p,shrink = .7)
plt.title('Wigner function of filtered state')
fid = fidelity(ssv,third)
print('fidelity with cat:',fid*fid)

fig, axes = plt.subplots(1, 4, figsize=(16,4))

bar0 = axes[0].bar(range(N_dim), third.diag())

lbl0 = axes[0].set_title("Third Filtered State")

lim0 = axes[0].set_xlim([-.5, 20])
bar1 = axes[1].bar(range(N_dim), second.diag())

lbl1 = axes[1].set_title("Second Filtered State")

lim1 = axes[1].set_xlim([-.5, 20])

bar2 = axes[2].bar(range(N_dim), first.diag())

lbl2 = axes[2].set_title("First Filtered State")

lim2 = axes[2].set_xlim([-.5, 20])

bar3 = axes[3].bar(range(N_dim), ket2dm(coherent(N_dim,delta)).diag())

lbl3 = axes[3].set_title("Initial Coherent State")

lim3 = axes[3].set_xlim([-.5, 20])
plt.show()
```

```python
fourth=Fock_Filter_prob(third,1,refl4,n4,eta,N_dim);
W4=wigner(fourth,xvec,xvec);

fig = plt.figure(figsize=(16,7))
# `ax` is a 3D-aware axis instance, because of the projection='3d' keyword argument to add_subplot
ax = fig.add_subplot(2, 2, 2)
p = ax.contourf(X, Y, W4, 20, cmap=cm.RdBu)
cb = fig.colorbar(p, shrink = 0.7)
# surface_plot with color grading and color bar
ax = fig.add_subplot(2, 2, 1, projection='3d')
p = ax.plot_surface(X, Y, W4, rstride=1, cstride=1, cmap=cm.RdBu, linewidth=0.5)
cb = fig.colorbar(p,shrink = .7)
plt.title('Wigner function of filtered state')
fid = fidelity(ssv,fourth)
print('fidelity with cat:',fid*fid)

fig, axes = plt.subplots(1, 4, figsize=(16,4))

bar0 = axes[0].bar(range(N_dim), fourth.diag())

lbl0 = axes[0].set_title("four-time Filtered State")

lim0 = axes[0].set_xlim([-.5, 20])
bar1 = axes[1].bar(range(N_dim), third.diag())

lbl1 = axes[1].set_title("three-time Filtered State")

lim1 = axes[1].set_xlim([-.5, 20])

bar2 = axes[2].bar(range(N_dim), second.diag())

lbl2 = axes[2].set_title("two-time Filtered State")

lim2 = axes[2].set_xlim([-.5, 20])

bar3 = axes[3].bar(range(N_dim), ket2dm(coherent(N_dim,delta)).diag())

lbl3 = axes[3].set_title("Initial Coherent State")

lim3 = axes[3].set_xlim([-.5, 20])
plt.show()
```

```python
final_state=D(fourth,beta*(-1))

W_final=wigner(final_state,xvec,xvec)

fig = plt.figure(figsize=(16,7))
# `ax` is a 3D-aware axis instance, because of the projection='3d' keyword argument to add_subplot
ax = fig.add_subplot(2, 2, 2)
p = ax.contourf(X, Y, W_ssv, 20, cmap=cm.RdBu)
cb = fig.colorbar(p, shrink = 0.7)
# surface_plot with color grading and color bar
ax = fig.add_subplot(2, 2, 1, projection='3d')
p = ax.plot_surface(X, Y, W_ssv, rstride=1, cstride=1, cmap=cm.RdBu, linewidth=0.5)
cb = fig.colorbar(p,shrink = .7)
plt.title('Wigner function of squeezed cat state')

fig = plt.figure(figsize=(16,7))
# `ax` is a 3D-aware axis instance, because of the projection='3d' keyword argument to add_subplot
ax = fig.add_subplot(2, 2, 2)
p = ax.contourf(X, Y, W_final, 20, cmap=cm.RdBu)
cb = fig.colorbar(p, shrink = 0.7)
# surface_plot with color grading and color bar
ax = fig.add_subplot(2, 2, 1, projection='3d')
p = ax.plot_surface(X, Y, W_final, rstride=1, cstride=1, cmap=cm.RdBu, linewidth=0.5)
cb = fig.colorbar(p,shrink = .7)
plt.title('Displaced Wigner function of |5>, |3> and |2> filtered coherent state')
fid = fidelity(ssv,final_state)
print('fidelity with ssv:',fid*fid)
plt.show()
fig, axes = plt.subplots(1, 5, figsize=(16,4))

bar0 = axes[0].bar(range(N_dim), second.diag())

lbl0 = axes[0].set_title("second Filtered")

lim0 = axes[0].set_xlim([-.5, 20])

bar1 = axes[1].bar(range(N_dim), third.diag())

lbl1 = axes[1].set_title("third Filtered")

lim1 = axes[1].set_xlim([-.5, 20])

bar2 = axes[2].bar(range(N_dim), fourth.diag())

lbl2 = axes[2].set_title("fourth Filtered State")

lim2 = axes[2].set_xlim([-.5, 20])

bar3 = axes[3].bar(range(N_dim), final_state.diag())

lbl3 = axes[3].set_title("Final")

lim3 = axes[3].set_xlim([-.5, 8])

bar4 = axes[4].bar(range(N_dim), ssv.diag())

lbl4 = axes[4].set_title("SSV")

lim4 = axes[4].set_xlim([-.5, 10])
plt.show()

```

```python
from mpl_toolkits.axes_grid1 import AxesGrid

def shiftedColorMap(cmap, start=1, midpoint=0, stop=0, name='shiftedcmap'):
    '''
    Function to offset the "center" of a colormap. Useful for
    data with a negative min and positive max and you want the
    middle of the colormap's dynamic range to be at zero.

    Input
    -----
      cmap : The matplotlib colormap to be altered
      start : Offset from lowest point in the colormap's range.
          Defaults to 0.0 (no lower offset). Should be between
          0.0 and `midpoint`.
      midpoint : The new center of the colormap. Defaults to
          0.5 (no shift). Should be between 0.0 and 1.0. In
          general, this should be  1 - vmax / (vmax + abs(vmin))
          For example if your data range from -15.0 to +5.0 and
          you want the center of the colormap at 0.0, `midpoint`
          should be set to  1 - 5/(5 + 15)) or 0.75
      stop : Offset from highest point in the colormap's range.
          Defaults to 1.0 (no upper offset). Should be between
          `midpoint` and 1.0.
    '''
    cdict = {
        'red': [],
        'green': [],
        'blue': [],
        'alpha': []
    }

    # regular index to compute the colors
    reg_index = np.linspace(start, stop, 257)

    # shifted index to match the data
    shift_index = np.hstack([
        np.linspace(0, midpoint, 128, endpoint=False),
        np.linspace(midpoint, 1, 129, endpoint=True)
    ])

    for ri, si in zip(reg_index, shift_index):
        r, g, b, a = cmap(ri)

        cdict['red'].append((si, r, r))
        cdict['green'].append((si, g, g))
        cdict['blue'].append((si, b, b))
        cdict['alpha'].append((si, a, a))

    newcmap = cm.colors.LinearSegmentedColormap(name, cdict)
    plt.register_cmap(cmap=newcmap)

    return newcmap
#from matplotlib import rc,rcParams
#from pylab import *
```

```python
#activate latex text rendering
#rc('text', usetex=True)
#rc('axes', linewidth=2)
#rc('font', weight='bold')
#rcParams['text.latex.preamble'] = [r'\usepackage{sfmath} \boldmath']

xvec = np.arange(-33.,33.)*5./40
yvec = np.arange(-35.,35.)*5./40
X,Y = np.meshgrid(xvec, xvec)  ##Some plotting params
W=wigner(final_state,xvec,xvec);
orig_cmap = cm.seismic

shifted_cmap1 = shiftedColorMap(orig_cmap, midpoint=1-(W.max()/(W.max()-W.min())), name='shifted')

#m = cm.ScalarMappable(cmap=shifted_cmap1)
#m.set_array([-1/math.pi,1/math.pi])
#m.set_clim(vmin=-1/math.pi,vmax=1/math.pi)

#m.set_array([-1,1])
#m.set_clim(vmin=-1,vmax=1)

fig = plt.figure(figsize=(8,5))
# `ax` is a 3D-aware axis instance, because of the projection='3d' keyword argument to add_subplot

ax1 = fig.add_subplot(1,1,1, projection='3d')
p = ax1.plot_surface(X, Y, W, rstride=1,cstride=1, cmap=shifted_cmap1, linewidth=0.5)
cb = fig.colorbar(p, shrink = 0.5)
cb.ax.tick_params(labelsize=12)

ax1.contour(X, Y, W, cmap=shifted_cmap1, linestyles="solid", offset=-0.3)
ax1.set_xlim([-4.5,4.5])
ax1.set_ylim([-4.5,4.5])
ax1.view_init(15, -40)
ax1.contour(X, Y, W,[0], zdir='y', cmap=cm.bwr, offset=4.5)
ax1.contour(X, Y, W,[0], zdir='x', cmap=cm.RdBu, offset=-4.5)
#plt.axis('off')
ax1.grid(b=False)
#plt.xlabel('P',fontsize=16,fontweight='heavy')
plt.xlabel('P',fontsize=16,fontweight='heavy')
plt.ylabel('X',fontsize=16,fontweight='heavy')
ax1.xaxis.set_tick_params(labelsize=12)
ax1.yaxis.set_tick_params(labelsize=12)
ax1.zaxis.set_tick_params(labelsize=12)

fig2 = plt.figure(figsize=(8,4))
ax1=fig2.add_subplot(1,2,1)
ax1.bar(range(N_dim),final_state.diag())
lim1 = ax1.set_xlim([-.5, 10.5])
lim1y = ax1.set_ylim([0,0.6])
plt.ylabel('Probability',fontsize='large')
plt.xlabel('Photon Number',fontsize='large')
plt.title('Approximate SSV')

ax2=fig2.add_subplot(1,2,2)
ax2.bar(range(N_dim),ssv.diag())
lim2 = ax2.set_xlim([-.5, 10.5])
lim2y = ax2.set_ylim([0,0.6])
plt.xlabel('Photon Number',fontsize='large')
plt.title('Ideal SSV')
#plt.subplots_adjust(wspace=0)
plt.show()
```

```python
'''Now plot the density matrix elements of the ideal state with that of
the approximate state resulting from photon-catalysis'''
s=np.abs((ssv.full()))
temp=np.delete(s,np.s_[10:],0)
state=np.delete(temp,np.s_[10:],1)

s2=np.abs((final_state.full()))
temp1=np.delete(s2,np.s_[10:],0)
state2=np.delete(temp1,np.s_[10:],1)

#plt.imshow(state)
fig = plt.figure(figsize=(6,4))
plt.pcolormesh(state, vmin=0.0, vmax=0.7)
cbar=plt.colorbar()
cbar.ax.tick_params(labelsize=14)
plt.tick_params(labelsize=14)
plt.show()

#plt.imshow(state2)
fig = plt.figure(figsize=(6,4))
plt.pcolormesh(state2, vmin=0.0, vmax=0.7)
cbar=plt.colorbar()
cbar.ax.tick_params(labelsize=14)
plt.tick_params(labelsize=14)
plt.show()

fig = plt.figure(figsize=(6,4))
plt.pcolormesh(np.abs(state2-state), vmin=0.0, vmax=0.7)
cbar=plt.colorbar()
cbar.ax.tick_params(labelsize=14)
plt.tick_params(labelsize=14)
plt.show()
```

```python
s=np.abs((ssv.full()))
temp=np.delete(s,np.s_[10:],0)
state=np.delete(temp,np.s_[10:],1)
#plt.imshow(state)
plt.pcolor(state, vmin=0.0, vmax=0.6)
plt.colorbar()
plt.show()
```

```python
s2=np.abs((final_state.full()))
temp1=np.delete(s2,np.s_[10:],0)
state2=np.delete(temp1,np.s_[10:],1)
#plt.imshow(state2)
plt.pcolor(state2, vmin=0.0, vmax=0.6)
plt.colorbar()
plt.show()
```

```python
plt.pcolor((np.abs(state2-state)), vmin=0.0, vmax=0.6)
#plt.pcolor(X, Y, f(data), cmap=cm, vmin=0.0, vmax=0.5)
plt.colorbar()
plt.show()
```

```python

```


# Variational Autoencoder for the not-MNIST dataset

We will use a probabilisitc non-linear generative model for the not-MNIST dataset. Unlike the localization example, we will train both the **generative model parameters** and the parameters of the **amortized variaitonal family**.

----

---

----

---

## ELBO lower-bound to $p(\mathbf{X})$

Where $\text{KL}(q_{\eta,\mathbf{x}}(\mathbf{z})||p(\mathbf{z}))$ is known in closed form since it is the KL divergence between two Gaussian pdfs:

\begin{align}
\text{KL}(q_{\eta,\mathbf{x}}(\mathbf{z})||p(\mathbf{z})) = \frac{1}{2} \left[\text{tr}\left(\text{diag}(\sigma_{\eta}(\mathbf{x}))\right)+\left(\mu_{\eta}(\mathbf{x})^T\mu_{\eta}(\mathbf{x})\right)-2-\log\det \left(\text{diag}(\sigma_{\eta}(\mathbf{x}))\right) \right]
\end{align}

## SGD optimization

- Sample a M-size minibatch of images.
- Sample $\mathbf{\epsilon}^{(i)}$ from $\mathcal{N}(\mathbf{0},\mathbf{I})$, $i=1,\ldots,M$.
- For $i=1,\ldots,M$, compute
\begin{align}
\mathbf{z}^{(i)} = \mu_\eta(\mathbf{x}^{(i)}) + \sqrt{\text{diag}(\sigma_\eta(\mathbf{x}^{(i)}))} \circ \mathbf{\epsilon}^{(i)}
\end{align}
- Compute gradients of
\begin{align}
\hat{\mathcal{L}}(\mathbf{X},\theta,\eta) =\sum_{i=1}^M \Big(\log  p_\theta(\mathbf{x}^{(i)}|\mathbf{z}^{(i)}) - \text{KL}(q_{\eta,\mathbf{x}^{(i)}}(\mathbf{z})||p(\mathbf{z}))\Big)
\end{align}
w.r.t. $\theta,\eta$
- Perform SGD update

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib as mpl
import tensorflow as tf
from matplotlib.offsetbox import OffsetImage, AnnotationBbox

# use seaborn plotting defaults
import seaborn as sns; sns.set()

%matplotlib inline
```

    /Users/olmos/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

## Loading MNIST database

```python
from sklearn.datasets import fetch_mldata
from sklearn.model_selection import train_test_split
mnist = fetch_mldata('MNIST original')
```

```python
images = mnist.data.astype(np.float32)
labels = mnist.target

train_dataset,test_dataset,train_labels,test_labels = train_test_split(images, labels, test_size=0.33, random_state=42)

train_dataset,valid_dataset,train_labels,valid_labels = train_test_split(train_dataset, train_labels, test_size=0.33, random_state=42)

#Generate a random index
num_figs=3
img_size = 28

index=np.random.randint(train_dataset.shape[0], size=num_figs)

for i in range(num_figs):
    plt.figure()
    plt.imshow(train_dataset[index[i],:].reshape([28,28]),cmap=plt.cm.gray)
    plt.title('Label:' + str(train_labels[index[i]]))
```

### TensorFlow Computation Graph and Loss Function

```python
z_dim = 2      #Latent Space

model_name = 'model1'   #In 'model1.py' we define the variational family

learning_rate = 1e-3
num_imgs = 50        #Number of samples generated from the generative model (for testing)
num_iter = 20000      #SGD iterations
period_plot = 1000
sigma_reconstruction = 0.1 #Reconstruction variance
batch_size = 200
dims = [batch_size,784]
```

```python
sess_VAE = tf.Graph()

with sess_VAE.as_default():

    print('[*] Importing model: ' + model_name)
    model = __import__(model_name)

    print('[*] Defining placeholders')

    inputX = tf.placeholder(tf.float32, shape=dims, name='x-input')

    print('[*] Defining the encoder')
    log_var, mean, sample_z, KL = model.encoder(inputX,z_dim,batch_size)

    print('[*] Defining the decoder')

    loglik,img_reconstruction = model.decoder(inputX,sample_z,sigma_reconstruction,dims[1])

    loss = -tf.reduce_mean(loglik - KL)

    optim = tf.train.AdamOptimizer(learning_rate).minimize(loss)

    print('[*] Defining Sample operation...')
    samples = model.new_samples(num_imgs, z_dim, dims[1])

    # Output dictionary -> Useful if computation graph is defined in a separate .py file

    tf_nodes = {}

    tf_nodes['X'] = inputX

    tf_nodes['mean'] = mean

    tf_nodes['logvar'] = log_var

    tf_nodes['KL'] = tf.reduce_mean(KL)

    tf_nodes['loglik'] = tf.reduce_mean(loglik)

    tf_nodes['img_reconst'] = img_reconstruction

    tf_nodes['optim'] = optim

    tf_nodes['samples'] = samples

```

    [*] Importing model: model1
    [*] Defining placeholders
    [*] Defining the encoder
    [*] Defining the decoder
    [*] Defining Sample operation...

## SGD optimization

```python

############ SGD Inference #####################################

with tf.Session(graph=sess_VAE) as session:

    # Add ops to save and restore all the variables.
    saver = tf.train.Saver()

    tf.global_variables_initializer().run()

    print('Training the VAE ...')

    for it in range(num_iter):

        # Pick an offset within the training data, which has been randomized.
        # Note: we could use better randomization across epochs.

        offset = (it * batch_size) % (train_labels.shape[0] - batch_size)

        # Generate a minibatch.

        batch_data = train_dataset[offset:(offset + batch_size), :]

        # Prepare a dictionary telling the session where to feed the minibatch.
        # The key of the dictionary is the placeholder node of the graph to be fed,
        # and the value is the numpy array to feed to it.

        feedDict = {tf_nodes['X'] : batch_data.reshape([-1,784])}

        _,loglik,KL= session.run([tf_nodes['optim'],tf_nodes['loglik'],tf_nodes['KL']],feedDict)

        if(it % period_plot ==0):

            print("It = %d, loglik = %.5f, KL = %.5f" %(it,loglik,KL))

    it = 0
    offset = (it * batch_size) % (train_labels.shape[0] - batch_size)

    batch_data_train = train_dataset[offset:(offset + batch_size), :]
    batch_data_test = test_dataset[offset:(offset + batch_size), :]

    #We compute the latent representation of batch images and their reconstruction
    feedDict = {tf_nodes['X'] : batch_data_train.reshape([-1,784])}
    z,reconstructions_train = session.run([tf_nodes['mean'],tf_nodes['img_reconst']], feed_dict=feedDict)

    feedDict = {tf_nodes['X'] : batch_data_test.reshape([-1,784])}
    z_test,reconstructions_test = session.run([tf_nodes['mean'],tf_nodes['img_reconst']], feed_dict=feedDict)

    samples = session.run(tf_nodes['samples'])

```

    Training the VAE ...
    It = 0, loglik = -29470248.00000, KL = 2.19100
    It = 1000, loglik = -15796347.00000, KL = 760.23541
    It = 2000, loglik = -14817572.00000, KL = 591.84204
    It = 3000, loglik = -13936310.00000, KL = 480.90906
    It = 4000, loglik = -13123023.00000, KL = 522.05859
    It = 5000, loglik = -12936282.00000, KL = 477.91397
    It = 6000, loglik = -12321731.00000, KL = 512.24341
    It = 7000, loglik = -12519340.00000, KL = 480.37546
    It = 8000, loglik = -12248061.00000, KL = 500.32953
    It = 9000, loglik = -12050847.00000, KL = 466.23477
    It = 10000, loglik = -12078188.00000, KL = 434.30856
    It = 11000, loglik = -11412385.00000, KL = 412.44876
    It = 12000, loglik = -11493222.00000, KL = 376.04962
    It = 13000, loglik = -11016027.00000, KL = 434.23788
    It = 14000, loglik = -10827702.00000, KL = 435.85867
    It = 15000, loglik = -11724360.00000, KL = 428.17090
    It = 16000, loglik = -11423675.00000, KL = 332.12469
    It = 17000, loglik = -10990386.00000, KL = 397.59558
    It = 18000, loglik = -10760596.00000, KL = 379.81311
    It = 19000, loglik = -10995891.00000, KL = 393.54282

### Lets plot some train images and their reconstruction

```python
#Lets plot train images and their reconstruction

n_plots_axis=5

f2, axarr2 = plt.subplots(n_plots_axis,2)

for i in range(n_plots_axis):

    axarr2[i,0].imshow(batch_data_train[i,:].reshape([28,28]),cmap='gray');
    axarr2[i,1].imshow(reconstructions_train[i,:].reshape([28,28]),cmap='gray')

```

### Lets plot latent representation of train images

```python
def plot_latent_space_with_images(ax,recons_images,z_samples):

    ax.clear()
    for i in range(len(recons_images)):
        im = OffsetImage(recons_images[i].reshape([28,28]), zoom=1,cmap='gray')
        ab = AnnotationBbox(im, z_samples[i],frameon=True)
        ax.add_artist(ab)
    ax.set_xlim(np.min(z_samples[:,0])-1,np.max(z_samples[:,0])+1)
    ax.set_ylim(np.min(z_samples[:,1])-1,np.max(z_samples[:,1])+1)
    ax.set_title('Latent space Z with Images')

plt.figure()
f_latent, ax_latent = plt.subplots(1,1,figsize=(8, 8))
plot_latent_space_with_images(ax_latent,reconstructions_train,z)
plt.rcParams["figure.figsize"] = [14,14]
```

```python
# Lets plot latent representation of test images

f3, axarr3 = plt.subplots(n_plots_axis,2)

for i in range(n_plots_axis):

    axarr3[i,0].imshow(batch_data_test[i,:].reshape([28,28]),cmap='gray');
    axarr3[i,1].imshow(reconstructions_test[i,:].reshape([28,28]),cmap='gray')

```

```python
# Lets plot samples of the model

n_plots_axis=5

f4, axarr4 = plt.subplots(n_plots_axis,2)

for i in range(n_plots_axis):

    axarr4[i,0].imshow(samples[i,:].reshape([28,28]),cmap='gray');
    axarr4[i,1].imshow(samples[i,:].reshape([28,28]),cmap='gray')

```

```python

```


# MTH9842 HW4
## Group L
## Yang Li, Jimmy Jusuf, Zhirong Zhang, Tiangang Zhang
## December 19, 2018

# Question.1

Let $\displaystyle{f(\mathbf{x})={-\!}\sum_{i=1}^n \log x_i}$ be the objective function, $\mathbf{x}\in \mathbb{R}^n$.

## (i)
We want to solve
$$
\min\ f(\mathbf{x}), \text{ subject to a single equality constraint } h(\mathbf{x})=\sum_{i=1}^n x_i -1 = 0
$$

Define Lagrange function

$$
L(\mathbf{x},\lambda) = f(\mathbf{x}) + \lambda\, h(\mathbf{x})
$$

Any optimal points $\mathbf{x}^*$ must satisfy KKT conditions:
\begin{align}
\nabla L_x = \nabla f(\mathbf{x}^*) + \lambda \nabla h(\mathbf{x}^*) = 0 \\
h(\mathbf{x}^*) = 0 \\
\lambda\geq 0 \\
\end{align}
We conclude
$$
\frac{\partial L}{\partial x_i} = 0
  \Rightarrow -\frac{1}{x_i} + \lambda = 0
  \Rightarrow x_i = \frac 1\lambda
$$
$$
h(\mathbf{x})=0 \Rightarrow \sum_{i=1}^n x_i = 1
  \Rightarrow \sum_{i=1}^n \frac1\lambda = 1
  \Rightarrow n\frac1\lambda = 1
  \Rightarrow \lambda = n
  \Rightarrow x_i = \frac 1n
$$

## (ii)
Given an optimization problem
$$
\begin{array}{ll}
  \text{minimize } & f(\mathbf{x})={-\!}\sum_{i=1}^n \log x_i \\
  \text{subject to } & \mathbf{Ax}-\mathbf{b}=0
\end{array}
$$
Similar to (i), define Lagrange function
$$
L(\mathbf{x},\boldsymbol\lambda) = f(\mathbf{x}) + (\mathbf{Ax} - \mathbf{b})^T\boldsymbol\lambda,
  \quad \mathbf{A}\in \mathbb{R}^{p\times n},
  \mathbf{b}\in\mathbb{R}^p,
  \boldsymbol\lambda\in\mathbb{R}^p
$$

The Lagrange dual function is
$$
q(\boldsymbol\lambda) = \inf_{\mathbf{x}\in\mathcal{D}}L(\mathbf{x},\boldsymbol\lambda)
$$

Holding $\boldsymbol\lambda$ fixed, this is a convex function of $x$. The infimum with respect
to $x$ is achieved when $\displaystyle{\frac{\partial L}{\partial x_i}}$ are zero.

Here, $A_{ji}$ is the $j$-th row, $i$-th col element in $\mathbf{A}$, $i\in 1\ldots n$
$$
\frac{\partial L}{\partial x_i} = 0
  \Rightarrow  -\frac{1}{x_i} + \lambda_j\sum_{j=1}^p A_{ji} = 0
  \Rightarrow x_i = \frac{1}{\lambda\sum_{j=1}^p A_{ji}} = \frac{1}{(\mathbf{A}^{\!T}\boldsymbol\lambda)_i}
$$

Also we have,
$$\mathbf{Ax} =
A\left(\begin{array}{c}\frac{1}{\lambda \sum_j A_{j1}}\\
\vdots\\
\frac{1}{\lambda \sum_j A_{jn}}\\
\end{array}\right) =
\left(
\begin{array}{c}\sum_i\frac{ A_{1i}}{\lambda \sum_j A_{ji}}\\
\vdots\\
\sum_i\frac{ A_{ni}}{\lambda \sum_j A_{ji}}\\
\end{array}\right)
$$
Thus we have
$$
\mathbf{x}^T\mathbf{A}^{\!T}\boldsymbol\lambda = \left(
\begin{array}{c}\sum_i\frac{ A_{1i}}{\lambda \sum_j A_{ji}}&
\cdots&
\sum_i\frac{ A_{ni}}{\lambda \sum_j A_{ji}}\\
\end{array}\right)
\left(\begin{array}{cc}\lambda\\
\vdots\\
\lambda \end{array}\right) = n
$$
Substituting into $q(\boldsymbol\lambda)$
$$
q(\boldsymbol\lambda) = \inf_{\mathbf{x}\in\mathcal{D}}L(\mathbf{x},\boldsymbol\lambda) =
  \inf_{\mathbf{x}\in\mathcal{D}}\left( -\!\sum_{i=1}^n \log x_i + (\mathbf{Ax} - \mathbf{b})^T\boldsymbol\lambda\right) =
  -\mathbf{b}^T\boldsymbol\lambda - \sum_{i = 1}^{n} \log\frac1{(\mathbf{A}^{\!T}\boldsymbol\lambda)_i} + n
$$
i.e. the dual problem is
$$
\text{maximize}\quad q(\boldsymbol\lambda) = -\mathbf{b}^T\boldsymbol\lambda +\sum_{i = 1}^{n}
  \log {(\mathbf{A}^{\!T}\boldsymbol\lambda)_i} + n
$$

## (iii)
Already solved in (ii)

## (iv)
By Newton's method, we need to calculate the matrix equation
$$
\begin{bmatrix}
\nabla^2 f &A^T\\
A& 0
\end{bmatrix}
\begin{bmatrix}
-\Delta x\\
\lambda
\end{bmatrix}
=
\begin{bmatrix}
-\nabla f(x)\\
0
\end{bmatrix}
$$
It's easy to see that

$$
\nabla f = (-\frac{1}{x_1},\cdots, -\frac{1}{x_n})^T
$$

$$
\nabla^2 f = \operatorname{diag}(\frac{1}{x_1^2},\cdots, \frac{1}{x_n^2})
$$

## (v)
To solve the dual problem, we use the following formula for gradient and hessian of
$q(\boldsymbol\lambda)$.

$$
\nabla_{\!\lambda}\,q=-\mathbf{b}+\mathbf{A}y \text{ where }
  y=(1/(\mathbf{A}^{\!T}\boldsymbol\lambda)_1,\ldots,
  1/(\mathbf{A}^{\!T}\boldsymbol\lambda)_n) \\
\nabla^2_{\!\lambda}\,q=-\mathbf{ADA}^{\!T}, \quad D \text{ is diagonal, with }
  D_{ii}=(\mathbf{A}^{\!T}\boldsymbol\lambda)_i^{-2}
$$

```python
import numpy as np
import pandas as pd

%matplotlib inline
import matplotlib.pyplot as plt

from numpy.linalg import inv

# global variables to keep track of search history
x_hist = []
fx_hist = []  # objective function value
dfx_hist = []  # derivative of objective function / gradient vector at x
hfx_hist = []  # second derivative of objective function / hessian matrix at x
dx_hist = []  # Newton step / search direction
epsilon_hist = []
```

```python
def vect(a):
    # turn input into a column vector, i.e. a.shape=(numrow,1)
    if isvect(a): return a  # do nothing if input is already a vector
    av=np.array(a)[:,None]
    #print(av.shape)
    assert av.shape==(len(a),1)
    return av

def isvect(x):
    # check if x is a vect
    try:
        assert isinstance(x, np.ndarray)
        if (x.ndim!=2): return False
        (nr,nc) = x.shape
        return nc==1
    except:
        return False

def sumcol(v):
    # input is a vector (assume column vector, but works with row vector too)
    # return the sum of elements
    assert isinstance(v, np.ndarray)
    return np.sum(v)
```

```python
a=vect([1,2,3])
b=np.array([4,5,6])
print(isvect(a), isvect(b)) # output: True False
```

    True False

```python
# IGNORE THIS
# Newton method for unconstrained problem
def NewtonUnconstrainedSolver(x0, alpha, beta, tol, objfun, objfun_grad, objfun_hess):
    assert np.isscalar(alpha) and np.isscalar(beta) and np.isscalar(tol)
    epsilon =  100 # get the Newton decrement
    gradtol = 1e-5
    count = 0
    x = x0
    while epsilon > tol and count < 500:
        count = count + 1
        funcval, gradvec, hessmat = objfun(x), objfun_grad(x), objfun_hess(x)
        print("x=", x, "\nf(x)=", funcval, "\ngrad(f(x))", gradvec, "\nhess(f(x))", hessmat)
        newton_step = -1*np.matmul(np.linalg.inv(hessmat), gradvec)
        print("newton_step", newton_step)
        descmag = np.squeeze(gradvec.T.dot(newton_step)) # check descent magnitude
        print("descent magnitude", descmag)
        # using backtracking line search
        back_ct=0
        t = 1
        # while -np.log(x + t * dx).sum() >= -np.log(x).sum() - alpha * t * dfx.T.dot(dx):
        while True:
            new_x = x+t*newton_step
            print("new_x", new_x)
            new_funcval = objfun(new_x)
            print(new_funcval, alpha, t, descmag)
            if new_funcval < (funcval-alpha*t*descmag): break
            t = t * beta
            back_ct+=1
            if back_ct>50: break
        if (abs(descmag)<gradtol): break
        # update the parameters
        last_x=x
        x = new_x
        fnew, gradvec, hessmat = new_funcval, objfun_grad(x), objfun_hess(x)
        # epsilon =  2*np.sqrt(-np.log(last_x).sum()+np.log(x).sum())
        epsilon =  2*np.sqrt(objfun(last_x)-objfun(x))
    return x,epsilon,count
```

```python
# IGNORE THIS
# class
class NewtonSolver:
    # Class attribute
    objfunc_val = 0;
    gradvec = 0;
    hess = 0;

    # Initializer / Instance attributes
    def __init__(self):
        None

    # instance method
    def objfunc(self, x):
        return "{} is {} years old".format(self.name, self.age)

    def grad(self, x):
        return "{} says {}".format(self.name, sound)
```

```python
# IGNORE THIS
# dual function optimizer
def nmdual(x0, A, b, alpha, beta, tol):
    def func(x):
        # implement the dual function
        (nr,nc) = A.shape
        atl=np.matmul(A.T,x)
        sumlog = sumcol(np.log(x))
        return -1*(-np.dot(b.T,x)+sumlog+nc)
    def grad(x):
        # return the gradient of objective function
        g=np.matmul(A, inv(np.matmul(A.T,x)))
        return -1*(vect(-b)+g)
    def hess(x):
        # return the hessian of the objective function
        atl=np.matmul(A.T,x)
        atlinv=inv(atl)
        Dii=np.diag(atlinv**2)
        D=np.diag(np.matmul(A.T,x))
        return -1*((-A @ D) @ A.T)
    # compute A' lambda
    #atl=np.matmul(A.T,x0)
    #atlinv=inv(atl)
    #Dii=np.diag(atlinv**2)
    return NewtonUnconstrainedSolver(x0, alpha, beta, tol, func, grad, hess)
```

```python
# IGNORE THIS
# test code for Newton unconstrained optimization problem
def nmprimal(x0, alpha, beta, tol):
    def func(x):
        # implement objective function f(x) = -sum(log(x_i))
        return sumcol(-np.log(x))
    def grad(x):
        # evaluate the gradient of objective function at point x
        return vect(-1/x)
    def hess(x):
        # evaluate the hessian of the objective function at point x
        return np.diagflat(1/x**2)
    return NewtonUnconstrainedSolver(x0, alpha, beta, tol, func, grad, hess)
```

```python
alpha = 0.1
beta = 0.5
tol = 0.0001
x0=vect([0.1,0.4,0.3])
A=np.hstack((vect([1,1,1]),vect([2,2,2]),vect([3,3,3])))
b=vect([1,1,1])
#nmdual(x0,alpha,beta,tol)
```

```python
def NewtonMethod(A, x0, alpha, beta, tol):
    """
    Implement Newton's method optimization with equality constraints.
    Input:
      A is the coefficient of the matrix that represents equality constraint (Ax-b)=0
      alpha, beta are parameters for backtracking line search
      tol, the algorithm terminates if |f(x)-f(last_x)| < tol
      Function f(), f_grad(), f_hess() represent the objective function to be minimized,
        the gradient and the hessian
    Output:
      x where the minimum is found
      epsilon the value of |f(x)-f(last_x)| from the final iteration
      count, the number of iterations
    """
    def f(x):
        # evaluate objective function at x
        return -np.log(x).sum()
    def f_grad(x):
        # return the gradient vector at f(x)
        return -1 / x
    def f_hess(x):
        # return the hessian matrix at f(x)
        return np.diag(1/x**2)
    (row, col) = A.shape
    x = x0
    H = f_hess(x) # H is hessian of f(x)
    upper = np.column_stack((H, A.T)) # [Hessian, A^T]
    lower = np.column_stack((A, np.zeros((row, row)))) # [A, 0]
    KKT = np.row_stack((upper, lower)) # this the the matrix on the left
    dfx = f_grad(x) # this is - nabla f(x)
    RHS = np.append(dfx, np.zeros(row)) # right hand side
    epsilon =  100 # get the Newton decrement
    count = 0
    # clear search history
    x_hist.clear()
    fx_hist.clear()  # objective function value
    dfx_hist.clear()  # derivative of objective function / gradient vector at x
    hfx_hist.clear()  # second derivative of objective function / hessian matrix at x
    dx_hist.clear()  # Newton step / search direction
    epsilon_hist.clear()
    while epsilon > tol and count < 500:

        count = count + 1
        # save the search paths
        x_hist.append(x)
        fx_hist.append(f(x))
        dfx_hist.append(dfx)
        hfx_hist.append(H)

        LHS = np.squeeze(np.array(inv(KKT).dot(RHS)))
        dx = -np.array(LHS)[:col]; dx_hist.append(dx)
        t = 1
        # using backtracking line search
        back_ct=0
        fval = f(x)
        print(fval)
        while f(x + t * dx) >= fval - alpha * t * dfx.T.dot(dx):
            t = t * beta
            back_ct+=1
            if back_ct>50:
                break
        # update the parameters
        last_x=x
        x = x + t * dx
        H = f_hess(x)
        upper = np.column_stack((H, A.T))
        lower = np.column_stack((A, np.zeros((row, row))))
        KKT = np.row_stack((upper, lower))
        dfx = f_grad(x)
        RHS = np.append(dfx, np.zeros(row))
        epsilon =  abs(f(last_x) - f(x)); epsilon_hist.append(epsilon)
        #print(dfx.T.dot(dx))
    return x,epsilon,count
```

```python
# IGNORE THIS
def NewtonMethodDual(A, b, x0, alpha, beta, tol):
    """
    Implement Newton's method optimization with equality constraints.
    Input:
      A is the coefficient of the matrix that represents equality constraint (Ax-b)=0
      alpha, beta are parameters for backtracking line search
      tol, the algorithm terminates if |f(x)-f(last_x)| < tol
      Function f(), f_grad(), f_hess() represent the objective function to be minimized,
        the gradient and the hessian
    Output:
      x where the minimum is found
      epsilon the value of |f(x)-f(last_x)| from the final iteration
      count, the number of iterations
    """
    def f(x):
        """ evaluate objective function at x """
        atx = A.T @ x
        btxm = -b.T @ x
        n = len(atx)
        return flipsign*(btxm+np.log(atx).sum()+n)
    def f_grad(x):
        """ return the gradient vector at f(x) """
        atx = A.T @ x
        y = 1/atx
        Ay = A @ y
        return flipsign*(-b+Ay)
    def f_hess(x):
        """ return the hessian matrix at f(x) """
        atx = A.T @ x
        y = 1/atx**2
        D = np.diagflat(y)
        ADAm = -A @ D @ A.T
        return flipsign*(ADAm)

    flipsign = -1
    (row, col) = A.shape
    x = x0
    H = f_hess(x) # H is hessian of f(x)
    upper = np.column_stack((H, A.T)) # [Hessian, A^T]
    lower = np.column_stack((A, np.zeros((row, row)))) # [A, 0]
    KKT = np.row_stack((upper, lower)) # this the the matrix on the left
    dfx = f_grad(x) # this is - nabla f(x)
    RHS = np.append(dfx, np.zeros(row)) # right hand side
    epsilon =  100 # get the Newton decrement
    count = 0
    # clear search history
    x_hist.clear()
    fx_hist.clear()  # objective function value
    dfx_hist.clear()  # derivative of objective function / gradient vector at x
    hfx_hist.clear()  # second derivative of objective function / hessian matrix at x
    dx_hist.clear()  # Newton step / search direction
    epsilon_hist.clear()
    while epsilon > tol and count < 500:

        count = count + 1
        # save the search paths
        x_hist.append(x)
        fx_hist.append(f(x))
        dfx_hist.append(dfx)
        hfx_hist.append(H)

        LHS = np.squeeze(np.array(inv(KKT).dot(RHS)))
        dx = -np.array(LHS)[:col]; dx_hist.append(dx)
        t = 1
        # using backtracking line search
        back_ct=0
        fval = f(x)
        while f(x + t * dx) >= fval - alpha * t * dfx.T.dot(dx):
            t = t * beta
            back_ct+=1
            if back_ct>50:
                break
        # update the parameters
        last_x=x
        x = x + t * dx
        H = f_hess(x)
        upper = np.column_stack((H, A.T))
        lower = np.column_stack((A, np.zeros((row, row))))
        KKT = np.row_stack((upper, lower))
        dfx = f_grad(x)
        RHS = np.append(dfx, np.zeros(row))
        epsilon =  abs(f(last_x) - f(x)); epsilon_hist.append(epsilon)
        #print(dfx.T.dot(dx))
    return x,epsilon,count
```

```python
# alpha = 0.1
# beta = 0.5
# tol = 1e-12
# A=np.array([[1,1,1,1,1]]) # define 1x5 matrix (p=1, n=5), single constraint sum(x_i)=1
# b=vect([1])
# x=np.array([2,2,2,2,2])
# atx=A.T @ l
# y=1/atx**2
# D=np.diagflat(y)
# -A @ D @ A.T
# #NewtonMethodDual(A, b, l, alpha, beta, tol)
# NewtonMethod(A, x, alpha, beta, tol)
```

```python
alpha = 0.1
beta = 0.5
tol = 1e-12
```

```python
# test with a small test case
# n=3 (dimension)
# p=1 (constrain sum(x_i)=1)
# x0 must start in the feasible set
x0=np.array([0.2,0.5,0.3])  # note the element-sum of x0 is 1
A=np.array([[1,1,1]])
b=np.array([[1]])
```

```python
%%time
NewtonMethod(A, x0, alpha, beta, tol)
```

    3.506557897319982
    3.3072267512534124
    3.2958912736500254
    3.295836866997819
    3.29583686600433
    Wall time: 15.6 ms

    (array([0.33333333, 0.33333333, 0.33333333]), 4.440892098500626e-16, 5)

```python
x_hist
```

    [array([0.2, 0.5, 0.3]),
     array([0.29473684, 0.34210526, 0.36315789]),
     array([0.3307705 , 0.33567068, 0.33355882]),
     array([0.33332553, 0.3333292 , 0.33334528]),
     array([0.33333333, 0.33333333, 0.33333333])]

```python
# note the constraint sum(x_i)=1 holds at all times during the search
[sum(x) for x in x_hist]
```

    [1.0, 1.0, 1.0, 1.0, 0.9999999999999998]

```python
ts=pd.Series(epsilon_hist)
plt=ts.plot(logy=True)
plt.set_xlabel('iteration')
plt.set_ylabel('epsilon')
ts
```

```python
# compute l1-norm of x_hist[0] with every elem of x_hist[0..n]
# the output should be practically zero since the algorithm ensures x stays feasible during the search
[sum(A@(x_hist[0] - x)) for x in x_hist]
```

    [0.0, 0.0, 0.0, 1.1102230246251565e-16, 2.220446049250313e-16]

```python
A1 = np.array([1 for i in range(500)])
```

```python
def cst(j,i):
    if j==i-1 or j==i:
        return 1
    else:
        return 0
```

```python
A2=np.array([[cst(j,i) for j in range(500)] for i in range(1,100)])
```

```python
A=np.row_stack((A1,A2))
```

```python
A
```

    array([[1, 1, 1, ..., 1, 1, 1],
           [1, 1, 0, ..., 0, 0, 0],
           [0, 1, 1, ..., 0, 0, 0],
           ...,
           [0, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0],
           [0, 0, 0, ..., 0, 0, 0]])

```python
x0 = np.array([0.5*i+1 for i in range(500)])
```

```python
x0
```

    array([  1. ,   1.5,   2. ,   2.5,   3. ,   3.5,   4. ,   4.5,   5. ,
             5.5,   6. ,   6.5,   7. ,   7.5,   8. ,   8.5,   9. ,   9.5,
            10. ,  10.5,  11. ,  11.5,  12. ,  12.5,  13. ,  13.5,  14. ,
            14.5,  15. ,  15.5,  16. ,  16.5,  17. ,  17.5,  18. ,  18.5,
            19. ,  19.5,  20. ,  20.5,  21. ,  21.5,  22. ,  22.5,  23. ,
            23.5,  24. ,  24.5,  25. ,  25.5,  26. ,  26.5,  27. ,  27.5,
            28. ,  28.5,  29. ,  29.5,  30. ,  30.5,  31. ,  31.5,  32. ,
            32.5,  33. ,  33.5,  34. ,  34.5,  35. ,  35.5,  36. ,  36.5,
            37. ,  37.5,  38. ,  38.5,  39. ,  39.5,  40. ,  40.5,  41. ,
            41.5,  42. ,  42.5,  43. ,  43.5,  44. ,  44.5,  45. ,  45.5,
            46. ,  46.5,  47. ,  47.5,  48. ,  48.5,  49. ,  49.5,  50. ,
            50.5,  51. ,  51.5,  52. ,  52.5,  53. ,  53.5,  54. ,  54.5,
            55. ,  55.5,  56. ,  56.5,  57. ,  57.5,  58. ,  58.5,  59. ,
            59.5,  60. ,  60.5,  61. ,  61.5,  62. ,  62.5,  63. ,  63.5,
            64. ,  64.5,  65. ,  65.5,  66. ,  66.5,  67. ,  67.5,  68. ,
            68.5,  69. ,  69.5,  70. ,  70.5,  71. ,  71.5,  72. ,  72.5,
            73. ,  73.5,  74. ,  74.5,  75. ,  75.5,  76. ,  76.5,  77. ,
            77.5,  78. ,  78.5,  79. ,  79.5,  80. ,  80.5,  81. ,  81.5,
            82. ,  82.5,  83. ,  83.5,  84. ,  84.5,  85. ,  85.5,  86. ,
            86.5,  87. ,  87.5,  88. ,  88.5,  89. ,  89.5,  90. ,  90.5,
            91. ,  91.5,  92. ,  92.5,  93. ,  93.5,  94. ,  94.5,  95. ,
            95.5,  96. ,  96.5,  97. ,  97.5,  98. ,  98.5,  99. ,  99.5,
           100. , 100.5, 101. , 101.5, 102. , 102.5, 103. , 103.5, 104. ,
           104.5, 105. , 105.5, 106. , 106.5, 107. , 107.5, 108. , 108.5,
           109. , 109.5, 110. , 110.5, 111. , 111.5, 112. , 112.5, 113. ,
           113.5, 114. , 114.5, 115. , 115.5, 116. , 116.5, 117. , 117.5,
           118. , 118.5, 119. , 119.5, 120. , 120.5, 121. , 121.5, 122. ,
           122.5, 123. , 123.5, 124. , 124.5, 125. , 125.5, 126. , 126.5,
           127. , 127.5, 128. , 128.5, 129. , 129.5, 130. , 130.5, 131. ,
           131.5, 132. , 132.5, 133. , 133.5, 134. , 134.5, 135. , 135.5,
           136. , 136.5, 137. , 137.5, 138. , 138.5, 139. , 139.5, 140. ,
           140.5, 141. , 141.5, 142. , 142.5, 143. , 143.5, 144. , 144.5,
           145. , 145.5, 146. , 146.5, 147. , 147.5, 148. , 148.5, 149. ,
           149.5, 150. , 150.5, 151. , 151.5, 152. , 152.5, 153. , 153.5,
           154. , 154.5, 155. , 155.5, 156. , 156.5, 157. , 157.5, 158. ,
           158.5, 159. , 159.5, 160. , 160.5, 161. , 161.5, 162. , 162.5,
           163. , 163.5, 164. , 164.5, 165. , 165.5, 166. , 166.5, 167. ,
           167.5, 168. , 168.5, 169. , 169.5, 170. , 170.5, 171. , 171.5,
           172. , 172.5, 173. , 173.5, 174. , 174.5, 175. , 175.5, 176. ,
           176.5, 177. , 177.5, 178. , 178.5, 179. , 179.5, 180. , 180.5,
           181. , 181.5, 182. , 182.5, 183. , 183.5, 184. , 184.5, 185. ,
           185.5, 186. , 186.5, 187. , 187.5, 188. , 188.5, 189. , 189.5,
           190. , 190.5, 191. , 191.5, 192. , 192.5, 193. , 193.5, 194. ,
           194.5, 195. , 195.5, 196. , 196.5, 197. , 197.5, 198. , 198.5,
           199. , 199.5, 200. , 200.5, 201. , 201.5, 202. , 202.5, 203. ,
           203.5, 204. , 204.5, 205. , 205.5, 206. , 206.5, 207. , 207.5,
           208. , 208.5, 209. , 209.5, 210. , 210.5, 211. , 211.5, 212. ,
           212.5, 213. , 213.5, 214. , 214.5, 215. , 215.5, 216. , 216.5,
           217. , 217.5, 218. , 218.5, 219. , 219.5, 220. , 220.5, 221. ,
           221.5, 222. , 222.5, 223. , 223.5, 224. , 224.5, 225. , 225.5,
           226. , 226.5, 227. , 227.5, 228. , 228.5, 229. , 229.5, 230. ,
           230.5, 231. , 231.5, 232. , 232.5, 233. , 233.5, 234. , 234.5,
           235. , 235.5, 236. , 236.5, 237. , 237.5, 238. , 238.5, 239. ,
           239.5, 240. , 240.5, 241. , 241.5, 242. , 242.5, 243. , 243.5,
           244. , 244.5, 245. , 245.5, 246. , 246.5, 247. , 247.5, 248. ,
           248.5, 249. , 249.5, 250. , 250.5])

```python
%%time
#res=NewtonMethod(A, x0, a, b, tol)
NewtonMethod(A, x0, alpha, beta, tol)
```

    -2270.9734742812684
    -2300.0694120185362
    -2305.033144809954
    -2305.3592866667277
    -2305.363440559174
    -2305.3634422023033
    Wall time: 234 ms

    (array([  1.25,   1.25,   2.25,   2.25,   3.25,   3.25,   4.25,   4.25,
              5.25,   5.25,   6.25,   6.25,   7.25,   7.25,   8.25,   8.25,
              9.25,   9.25,  10.25,  10.25,  11.25,  11.25,  12.25,  12.25,
             13.25,  13.25,  14.25,  14.25,  15.25,  15.25,  16.25,  16.25,
             17.25,  17.25,  18.25,  18.25,  19.25,  19.25,  20.25,  20.25,
             21.25,  21.25,  22.25,  22.25,  23.25,  23.25,  24.25,  24.25,
             25.25,  25.25,  26.25,  26.25,  27.25,  27.25,  28.25,  28.25,
             29.25,  29.25,  30.25,  30.25,  31.25,  31.25,  32.25,  32.25,
             33.25,  33.25,  34.25,  34.25,  35.25,  35.25,  36.25,  36.25,
             37.25,  37.25,  38.25,  38.25,  39.25,  39.25,  40.25,  40.25,
             41.25,  41.25,  42.25,  42.25,  43.25,  43.25,  44.25,  44.25,
             45.25,  45.25,  46.25,  46.25,  47.25,  47.25,  48.25,  48.25,
             49.25,  49.25,  50.25,  50.25, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75, 150.75,
            150.75, 150.75, 150.75, 150.75]), 4.547473508864641e-13, 6)

```python
ts=pd.Series(epsilon_hist)
plt=ts.plot(logy=True)
plt.set_xlabel('iteration')
plt.set_ylabel('epsilon')
ts
```

```python
# to show the algorithm enforces the constraints correctly,
# we show l1-norm of x_hist[0] with every elem of x_hist[0..n] are zero
[sum(A@(x_hist[0] - x)) for x in x_hist]
```

    [0.0,
     3.099076550938662e-12,
     5.9821037012852685e-12,
     2.2035706592760107e-12,
     1.0793810290010697e-11,
     1.5482060078397808e-11]

# Question.2

## (a)

First, show the equivelent between these 4 conditions.
___
* Condition 1 is equivalent to Condition 2

Condition 1 $\rightarrow$ Condition 2

Suppose $x \in \mathcal N(A) \cap \mathcal N(H)$ and $x \neq 0$. In condition 2, we have  $Ax = 0\, x \neq 0$. Since $x \in \mathcal N(A) \cap \mathcal N(H)$, we have $x^THx = 0$, this contradict to to second statement.

Condition 2 $\rightarrow$ Condition 1

If there is an $x$ s.t. $Ax = 0\, x \neq 0, x^THX = 0$. Because $H$ is semipositive definate, we must have $Hx = 0$, i.e.   $x \in \mathcal N(A) \cap \mathcal N(H)$
___
* Condition 2 is equivalent to Condition 3

If $Ax = 0, x \neq 0$, then because $\mathcal R(F) = \mathcal N (A)$, we have $x = Fz$ and $z \neq 0$. Then because of 2 we have $x^T Hx = z^TF^THFZ > 0$

___
* Condition 2 is equivalent to Condition 4
If we have the condition 2. then

$$
x^T(H + A^TA) = x^THx + \vert\vert A^Tx\vert\vert^2_2 > 0
$$
for all nonzero x, sot the condition 4 holds for $Q = I$.
If the fourth condition hols with general semipositive definate form of $Q$.
$$
x^T(H + A^TQA) = x^THx + x^TA^TQAx > 0
$$
for all nonzero $x$, Therefore if $Ax = 0\, x \neq 0$, we must have $x\neq 0$
___
Second show these four conditions are equivalent to nonsigularity of KKT matrix.

Suppose $x\neq 0$, s.t.$Ax = 0, Hx = 0$
$$
\begin{bmatrix}
H & A^T\\
A & 0
\end{bmatrix}
\begin{bmatrix}
x\\
0
\end{bmatrix} = 0$$

If the KKT matrix is singular, $x,z$ are not both zero, and we have
$$
\begin{bmatrix}
H & A^T\\
A & 0
\end{bmatrix}
\begin{bmatrix}
x\\
z
\end{bmatrix} = 0$$

This leads to $Hx + A^Tz = 0$, and $Ax = 0$.
from the first equation, we have
$$
x^THx + x^TA^Tz = 0.
$$
plug in the second equation.
we have $x^THx = 0$, this contradicts second condition unless $x= 0$.

## (b)
From previous question, we have $H + A^TA$ is positive definate matrix. Then we must hvae $R \in \mathbb R^{n\times n}$, s.t.
$$
R^T(H + A^TA)R = I
$$
Apply SVD to $AR$, $AR = U\Sigma V_1^T$, where $\Sigma = diag(\sigma_1,\cdots, \sigma_p)$. Denote $V_2 \in \mathbb R^{n\times(n-p)}$, s.t.
$$
V = \left[V_1\quad V_2\right]
$$
$V_2$ is orthogonal, and let
$$
S = \left[\Sigma\quad 0\right] \in \mathbb R^{p\times n}
$$
We have $AR = USV^T$, then
$$
V^TR^T(H + A^TA)RV = V^TR^THRV + S^TS = I
$$
Becasue $V^TR^THRV = I - S^TS$ is diagonal matrix, then denote it as $D$
$$
D = V^TR^THRV = diag(1-\sigma_1^2, \cdots, 1-\sigma_p^2, 1, \cdots, 1)
$$
Apply the transform to KKT matrix, we have

$$
\begin{bmatrix}
V^TR^T & 0\\
0 & U
\end{bmatrix}
\begin{bmatrix}
H & A^T\\
A & 0
\end{bmatrix}
\begin{bmatrix}
RV & 0\\
0 & U
\end{bmatrix}
=
\begin{bmatrix}
D & S^T\\
S & 0
\end{bmatrix}
$$
Apply the permutation to hte matrix on the right gives a block diaonal matrix with n diagonal blocks.
$$
\begin{bmatrix}
\lambda_i & \sigma_i\\
\sigma_i & 0
\end{bmatrix}
$$
where $i = 1,2,\cdots, p$,

and $\lambda_i = 1$ for $i = p +1, \cdots, n$

This matrix have eiggenvalues of $\frac{\lambda_i \pm \sqrt{\lambda_i^2 + 4\sigma_i^2}}{2}$, i.e. one eigen value is positive, another is negative.

In total, there are n positive evalues and p negative evalues.


```python
import numpy as np
import matplotlib.pyplot as plt
import math
from numpy.linalg import inv
from numpy import matmul as mm
import scipy
from scipy.linalg import lu_factor,lu_solve

x = np.linspace(-4,2,20)
y = (x+1)**3+10*np.random.normal(0,1,20)
def polynomialFitLU(x,y,order=3):
    for i in range(order+1):
        if i == 0:
            Phi = x**i
        else:
            Phi = np.vstack((Phi,x**i))
    Phi = Phi.T
    if order == 0:
        Phi = Phi.reshape(-1,1)
    lu,piv = lu_factor(mm(Phi.T,Phi))
    tmp = lu_solve((lu,piv),Phi.T)
    w = mm(tmp,y)
    f = mm(Phi,w)
    dif = y-f
    err = mm(dif.T,dif)
    return f,err,w,Phi
```

```python
errlistLU = []
plt.figure(figsize=(20,20))
for i in range(21):
    plt.subplot(7,3,i+1)
    f,err,w,Phi = polynomialFitLU(x,y,i)
    errlistLU.append(err)
    plt.plot(x,y,'b.')
    plt.plot(x,f,'r-')
    plt.title('Order '+str(i)+': '+str(err))
```

The sum of squared errors is a convex function of $\mathbf{w}$:

$$E(\mathbf{w})=(\mathbf{y}-\mathbf{\Phi w})^\top(\mathbf{y}-\mathbf{\Phi w})$$

The weight vector is

$$\mathbf{\hat{w}}=(\mathbf{\Phi^\top\Phi})^{-1}\mathbf{\Phi^\top y}$$

### Observation noise
The observations are noisy:
$$y_n = f_\mathbf{w}(x_n)+\epsilon_n$$

The noise can be characterized with a probability density function, for example, a Gaussian density function, $\epsilon_n\sim\mathcal{N}(0,\sigma^2_{noise})$,
$$p(\epsilon_n)=\frac{1}{\sqrt{2\pi\sigma^2_{noise}}}\exp\bigg(-\frac{\epsilon^2_n}{2\sigma^2_{noise}}\bigg)$$

The independent noise terms can be stacked as $\mathbf{\epsilon}=[\epsilon_1,...,\epsilon_N]^\top$,

\begin{align}
\mathbf{\epsilon}&\sim\mathcal{N}(\mathbf{\epsilon};0,\sigma^2_{noise}\mathbf{I})\\
p(\mathbf{\epsilon})&=\prod^N_{i=1}p(\epsilon_n)=\bigg(\frac{1}{\sqrt{2\pi\sigma^2_{noise}}}\bigg)^N\exp(-\frac{\epsilon^\top\epsilon}{2\sigma^2_{noise}})
\end{align}

Given that $\mathbf{y}=\mathbf{f}+\mathbf{\epsilon}$,

\begin{align}
p(\epsilon) &= p(\mathbf{y}|\mathbf{f},\sigma^2_{noise}) = \mathcal{N}(\mathbf{y};\mathbf{f},\sigma^2_{noise}) \\
&= \bigg(\frac{1}{\sqrt{2\pi\sigma^2_{noise}}}\bigg)^N\exp\bigg(-\frac{\|\mathbf{y}-\mathbf{f}\|^2}{2\sigma^2_{noise}}\bigg) \\
&= \bigg(\frac{1}{\sqrt{2\pi\sigma^2_{noise}}}\bigg)^N\exp\bigg(-\frac{E(\mathbf{w})}{2\sigma^2_{noise}}\bigg)
\end{align}

### Likelihood
The likelihood of the parameters is the probability of the data given parameters.

- $p(\mathbf{y}|\mathbf{w},\sigma^2_{noise})$ is the probability of the observed data given the weights.
- $\mathcal{L}(\mathbf{y}|\mathbf{w},\sigma^2_{noise})$ is the likelihood of the weights.

### Maximum likelihood
Fit the model weights to the data by maximizing the likelihood:

$$\mathbf{\hat{w}}=\arg\max\mathcal{L}(\mathbf{w})=\arg\max\exp\bigg(-\frac{E(\mathbf{w})}{2\sigma^2_{noise}}\bigg)=\arg\min E(\mathbf{w})$$

With an additive Gaussian independent noise model, the **maximum likelihood** and the **least squares** solutions are the same.

```python
print(errlistLU)
print(w)
```

    [2082.9435443058337, 915.5179868728524, 852.0086960680927, 779.9021317868293, 761.0539882151551, 662.4078175155137, 643.8318455004589, 529.4502292781594, 490.29415012657256, 471.5180097213798, 322.2365737351904, 321.91181712781224, 300.01857072519704, 298.67675984945015, 289.20164220557547, 284.06943927914, 267.1003813972985, 266.28824536898287, 294.06906171749114, 296.3996717705036, 194.073186960229]
    [ 6.39773752e+00  1.60476718e+01 -1.99027687e+01 -1.24549999e+02
      1.12954352e+02  3.30341907e+02 -1.20378571e+02 -3.84728228e+02
     -8.16607630e+00  2.01465464e+02  5.35819868e+01 -4.21247674e+01
     -2.01745910e+01  1.16151911e+00  1.91175012e+00  2.98135810e-01
      5.14467134e-02  2.69740281e-02  4.53298263e-03 -8.50544471e-05
     -4.92070226e-05]

Assume $\sigma_{noise}=10$,

```python
np.exp(-np.array(errlistLU)/100/2)
```

    array([2.99878661e-05, 1.02796411e-02, 1.41216884e-02, 2.02518191e-02,
           2.22531893e-02, 3.64417899e-02, 3.99886655e-02, 7.08456901e-02,
           8.61667634e-02, 9.46480458e-02, 1.99651313e-01, 1.99975767e-01,
           2.23109443e-01, 2.24611329e-01, 2.35508515e-01, 2.41630109e-01,
           2.63026128e-01, 2.64096364e-01, 2.29846104e-01, 2.27183241e-01,
           3.78944344e-01])

```python
plt.plot(np.arange(21),np.exp(-np.array(errlistLU)/100/2))
plt.title('Likelihood')
```

```python

```

Probabilistic Machine Learning/Introduction to Probabilistic Machine Learning/.ipynb_checkpoints/Likelihood and the concept of noise-checkpoint.ipynb